[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509606#comment-15509606
]
Sergio Bossa commented on CASSANDRA-12039:
--
Thanks for helping and reviewing [~beobal]!
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
> Fix For: 3.10
>
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506862#comment-15506862
]
Sergio Bossa commented on CASSANDRA-12039:
--
bq. The patch should only go into 3.x though.
Fine for me.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
> Fix For: 3.x
>
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505238#comment-15505238
]
Aleksey Yeschenko commented on CASSANDRA-12039:
---
CI results look clean to me (with one failure that I'm pretty sure is
unrelated). The patch should only go into 3.x though.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
> Fix For: 3.x
>
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488110#comment-15488110
]
Sam Tunnicliffe commented on CASSANDRA-12039:
-
Ok, I've rebased my dtest branch as there was some restructuring of the
{{tools}} module since last update. For simplicity's sake, I've opened a [new
PR|https://github.com/riptano/cassandra-dtest/pull/1316] based on my branch and
will commit this as soon as the dtest get a +1.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449274#comment-15449274
]
Sam Tunnicliffe commented on CASSANDRA-12039:
-
Already done :)
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449267#comment-15449267
]
Sergio Bossa commented on CASSANDRA-12039:
--
bq. I just pushed an update to my dtest branch which changes resume_test to use
byteman
Yay! I didn't know it was possible to use Byteman, great job. Could you just
remove the {{timeout}} parameter from {{_base_test}}?
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449255#comment-15449255
]
Sergio Bossa commented on CASSANDRA-12039:
--
Looks like we're in an impasse here: no one can see what's wrong with the
patch, or the tests, and the test infrastructure doesn't provide enough aid to
properly debug the problem. The test will not magically fix itself, so we
either remove it or pause this issue until the test infrastructure is improved
to _at least_ allow to take thread dumps of hanging nodes. Thoughts?
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449240#comment-15449240
]
Sam Tunnicliffe commented on CASSANDRA-12039:
-
I just pushed an update to [my dtest
branch|https://github.com/beobal/cassandra-dtest/commits/CASSANDRA-12039] which
changes {{resume_test}} to use byteman to inject a failure into the bootstrap
process, as {{bootstrap_test:TestBootstrap.resumable_bootstrap_test}} now does.
This seems to have stabilized the test (at least running locally) and also has
the side effect of reducing the amount of data needed during {{_base_test}},
which speeds up the run considerably.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449224#comment-15449224
]
Philip Thompson commented on CASSANDRA-12039:
-
I cannot see anything in the test that would cause the issue
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448850#comment-15448850
]
Sergio Bossa commented on CASSANDRA-12039:
--
Ping [~beobal], [~philipthompson].
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437331#comment-15437331
]
Sergio Bossa commented on CASSANDRA-12039:
--
I tried to debug it but I couldn't find anything relevant in the logs, and the
test actually hangs up locally for me most of the time.
The test looks correct and I have no idea why the node hangs out of the blue: I
tried taking some thread dumps but I couldn't find the log they end up in. The
only thing I can think of is to remove the test, thoughts?
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436954#comment-15436954
]
Philip Thompson commented on CASSANDRA-12039:
-
Thanks for catching the bad test reports. I've looked at the jobs, and it's a
problem with the new dtest, {{resume_test}}. It's hanging, or otherwise not
finishing in a way that causes the machine it ran on to not report test results.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434713#comment-15434713
]
Sam Tunnicliffe commented on CASSANDRA-12039:
-
I've rebased both the C* and dtest branches, but I'm still seeing some issues
on the CI runs. The dtest failure on the trunk branch looks unrelated and I
can't repro locally, but the one on the 3.9 branch looks relevant. I've tried
to repro that but it consistently times out for me when run locally.
I'm also concerned about the test counts; the 3.0, 3.9 & trunk runs are
reporting 692, 863 & 669 tests respectively compared to the mainline jobs
reporting 858. As of last night when I kicked off the last run, the custom
dtest branch was completely up to date with riptano/master so I'm not sure how
to account for those discrepancies. I know we had some setup issues recently
which presented like this, but I thought those were all fixed now. Any ideas
[~philipthompson]?
||branch||testall||dtest||
|[12039-3.0|https://github.com/beobal/cassandra/tree/12039-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.0-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.0-dtest]|
|[12039-3.9|https://github.com/beobal/cassandra/tree/12039-3.9]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.9-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-3.9-dtest]|
|[12039-trunk|https://github.com/beobal/cassandra/tree/12039-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-12039-trunk-dtest]|
dtest branch:
[CASSANDRA-12039|https://github.com/beobal/cassandra-dtest/tree/CASSANDRA-12039]
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421549#comment-15421549
]
Sergio Bossa commented on CASSANDRA-12039:
--
With [~philipthompson]'s suggestion, dtests now pass, except some apparently
unrelated failures:
https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-dtest/17/#showFailuresLink
Are we good to merge?
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420505#comment-15420505
]
Philip Thompson commented on CASSANDRA-12039:
-
I tried running it with debug output on the node that's failing to start, and
all that did was make it pass every time:
https://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/246/
This makes me think bumping the start timeout on that node should be
sufficient, or at the very least, that the issue is not in your
C* patch
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418696#comment-15418696
]
Sergio Bossa commented on CASSANDRA-12039:
--
For the record, the failing dtest passes for me locally /cc [~philipthompson]
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415034#comment-15415034
]
Sergio Bossa commented on CASSANDRA-12039:
--
I've rebased both the C* and dtests patches, but dtests keep failing in strange
ways, i.e.:
https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-dtest/15/testReport/junit/secondary_indexes_test/TestPreJoinCallback/manual_join_test/
I believe it's some sort of instability in the test run, rather than in the
test itself, any suggestions how to move forward?
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407754#comment-15407754
]
Sergio Bossa commented on CASSANDRA-12039:
--
bq. your new dtests look good to me, would you mind opening a PR for them
please?
Done: https://github.com/riptano/cassandra-dtest/pull/1170
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407702#comment-15407702
]
Sam Tunnicliffe commented on CASSANDRA-12039:
-
The patch LGTM, but there seem to have been a few issues with CI. It looks like
environmental issues have prevented the testall job from running successfully,
and there are some failures on the latest dtest run. Two of the three failures
there are for bugs that have been fixed recently, so I've rebased on latest 3.0
and re-started CI. I'll commit when I get a clean run.
[~sbtourist] your new dtests look good to me, would you mind opening a PR for
them please?
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404586#comment-15404586
]
Sergio Bossa commented on CASSANDRA-12039:
--
[~beobal],
I've updated the patch to make it work when joining after survey mode as
described above, and added dtests for all cases (hopefully), here are the
updated links:
|[3.0
patch|https://github.com/apache/cassandra/compare/cassandra-3.0...sbtourist:CASSANDRA-12039-3.0?expand=1]|
[dtests
patch|https://github.com/riptano/cassandra-dtest/compare/master...sbtourist:CASSANDRA-12039?expand=1]|
[dtests
run|https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-dtest/]|[testall
run|https://cassci.datastax.com/view/Dev/view/sbtourist/job/sbtourist-CASSANDRA-12039-3.0-testall/]|
Regarding the second problem described above, moving the "ring wait" and
"migrations wait" outside of the bootstrap block is probably too invasive for
this narrow case, so I ended up ignoring it, assuming that if the 2i is
migrated after the node joins, the callback will have nothing to do (basically
treating it the same way as if a new table were created).
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404570#comment-15404570
]
Sergio Bossa commented on CASSANDRA-12039:
--
Thanks.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404320#comment-15404320
]
Brandon Williams commented on CASSANDRA-12039:
--
Write survey mode works exactly like bootstrap, it just never sets the
completed flag, so nodes continue to forward 'extra' writes to it, but then
later if you decide the node should join you can do so with nodetool. Or if
you decide it's bad, you can just kill it, since it never became a full member
of the ring.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403873#comment-15403873
]
Sergio Bossa commented on CASSANDRA-12039:
--
I'm writing dtests for all cases requiring a call to the pre-join task, and I
noticed the following:
1) There's a bootstrap test ({{bootstrap_on_write_survey_test}}) that
explicitly verifies the bootstrap state is left in progress after survey mode,
so it seems to me we can actually rely on it to signal the bootstrap status on
joining; pinging again [~brandon.williams] for feedback.
2) There's one more problem related to calling the pre-join task when joining
*without* bootstrapping; in such case, the node doesn't wait for the migrations
to complete (that is, {{MigrationManager#waitUntilReadyForBootstrap()}} is
called only if bootstrapping), so it misses to wait for the new schema
containing the column family, and no task is invoked. Should we run the
{{MigrationManager}} checks regardless if the node is actually streaming or not
(I'd say so)? /cc [~beobal] [~iamaleksey]
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397457#comment-15397457
]
Sergio Bossa commented on CASSANDRA-12039:
--
[~beobal],
I've pushed one more commit addressing your latest points, and I'll look into
writing any dtests. Also:
bq. One thing that we should note is that when pre-join tasks are executed as a
result of manually joining (i.e. start in write survey mode, then join via
nodetool/JMX), the value of the bootstrap flag is always false. Obviously, this
might not always be accurate as a node can perfectly well start up in survey
mode but also perform bootstrap before an operator joins it. Short of exposing
the overload of joinRing and extending the nodetool join command, I'm not sure
there's any practical solution to this.
I think we could actually leverage the bootstrap state, which is left pending
at {{IN_PROGRESS}} during write survey and then set at {{COMPLETED}} when
finally joining. But, I'm not really sure why the write survey mode works that
way, that is why the bootstrap state is left in progress, and I don't want to
rely on some unintentional behaviour: any ideas, [~brandon.williams]?
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391866#comment-15391866
]
Sam Tunnicliffe commented on CASSANDRA-12039:
-
Thanks, the updated patch is looking good. If you amend slightly the new log
statement in {{SIM}} to include the value being passed in, we can properly
check that in a dtest. Otherwise, modulo the couple of nits below I think we'll
be good to go once we get some dtests in place.
One thing that we should note is that when pre-join tasks are executed as a
result of manually joining (i.e. start in write survey mode, then join via
nodetool/JMX), the value of the bootstrap flag is always false. Obviously, this
might not always be accurate as a node can perfectly well start up in survey
mode but also perform bootstrap before an operator joins it. Short of exposing
the overload of {{joinRing}} and extending the nodetool join command, I'm not
sure there's any practical solution to this. However, I guess this is quite a
niche scenario, so I suggest that clearly documenting this limitation in
{{NEWS.txt}} is sufficient. Would you mind adding an entry to {{NEWS.txt}}
please?
Nits:
The filter condition in {{StorageService::executePreJoinTasks}} can be
streamlined to:
{code}
.filter(cfs ->
Schema.instance.getUserKeyspaces().contains(cfs.keyspace.getName()))
{code}
{{SIM::executeAllBlocking}} handles null {{Callable}} correctly, so the default
impl of {{getPreJoinTask}} can just return {{null}} rather than a task which
returns {{null}}.
There's a couple of other minor formatting things, but I can fix those on
commit.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385707#comment-15385707
]
Sergio Bossa commented on CASSANDRA-12039:
--
[~beobal], new version pushed, fee free to review again.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385680#comment-15385680
]
Sergio Bossa commented on CASSANDRA-12039:
--
bq. A failed bootstrap causes the node to remain in the JOINING state until
successfully resumed. When it does finally succeed though, it follows the same
code path as leaving survey mode, so fixing that should also take care of this.
Got it, good point.
bq. The impl that you added to CassandraIndex should be moved to Index, making
it a default method
Of course, and that's to preserve backward compatibility with other index
implementations. Let me fix that.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385641#comment-15385641
]
Sam Tunnicliffe commented on CASSANDRA-12039:
-
bq.I think this is ok, as the node will have already joined the ring, and this
is mainly a pre-join callback.
A failed bootstrap causes the node to remain in the {{JOINING}} state until
successfully resumed. When it does finally succeed though, it follows the same
code path as leaving survey mode, so fixing that should also take care of this.
bq.I provided an inline "null" implementation similar to what happens with
other get*Task methods, what should I do differently?
The impl that you added to {{CassandraIndex}} should be moved to {{Index}},
making it a {{default}} method on the iface so that adding that same
implementation to every custom 2i is not necessary.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12039) Add an index callback to be notified post bootstrap and before joining the ring
[
https://issues.apache.org/jira/browse/CASSANDRA-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384752#comment-15384752
]
Sergio Bossa commented on CASSANDRA-12039:
--
[~beobal],
thanks for your comments, see my answers below:
bq. Pre-join tasks are not executed if a node is started in write survey mode
and then fully joins the ring later.
Good point, I can fix this.
bq. If bootstrap fails and is subsequently resumed, pre-join tasks are not
executed on its completion.
I think this is ok, as the node will have already joined the ring, and this is
mainly a pre-join callback.
bq. Index::getPreJoinTask should have a default no-op implementation
I'm not sure I get this: I provided an inline "null" implementation similar to
what happens with other {{get*Task}} methods, what should I do differently?
bq. I think it would be sufficient to add some debug logging to
StorageService::executePreJoinTasks and check for that in the node logs.
Sure I can do that.
> Add an index callback to be notified post bootstrap and before joining the
> ring
> ---
>
> Key: CASSANDRA-12039
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12039
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Sergio Bossa
>Assignee: Sergio Bossa
>
> Custom index implementations might need to be notified when the node finishes
> bootstrapping in order to execute some blocking tasks before the node itself
> goes into NORMAL state.
> This is a proposal to add such functionality, which should roughly require
> the following:
> 1) Add a {{getPostBootstrapTask}} callback to the {{Index}} interface.
> 2) Add an {{executePostBootstrapBlockingTasks}} method to
> {{SecondaryIndexManager}} calling into the previously mentioned callback.
> 3) Hook that into {{StorageService#joinTokenRing}}.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
