[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278376#comment-15278376
]
Paulo Motta commented on CASSANDRA-11279:
-
got it, thanks for the clarification!
+1, marking as ready to commit.
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Marcus Eriksson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278257#comment-15278257
]
Marcus Eriksson commented on CASSANDRA-11279:
-
The new bootstrapping node is balanced, but the asserts check that the nodes we
take ranges from are still balanced after running cleanup, and this is not as
easy to do, and that is the reason we need CASSANDRA-11643
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Marcus Eriksson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278248#comment-15278248
]
Paulo Motta commented on CASSANDRA-11279:
-
shouldn't disks be balanced (with a 20% tolerance) for a random distribution of
workload + tokens? At least we could guarantee that, not much, but better than
nothing :)
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Marcus Eriksson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278241#comment-15278241
]
Marcus Eriksson commented on CASSANDRA-11279:
-
[~pauloricardomg] I don't think that will improve things? The randomness will
still make those asserts fail at times, or am I missing something?
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Marcus Eriksson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271726#comment-15271726
]
Paulo Motta commented on CASSANDRA-11279:
-
Should we maybe keep the asserts while unsetting
{{allocate_tokens_for_keyspace}} to test that disks are balanced for the random
allocation bootstrap case?
We can extend these after CASSANDRA-11643 to include the balanced bootstrap
case.
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Marcus Eriksson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211855#comment-15211855
]
Philip Thompson commented on CASSANDRA-11279:
-
[~krummas], even bumping it up to 1 million keys does not solve the problem.
Not of the other disk balance tests flake at that level though:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/41/
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Philip Thompson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208468#comment-15208468
]
Marcus Eriksson commented on CASSANDRA-11279:
-
[~philipthompson] yeah since we only write 10k keys it could be, can you try it?
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Philip Thompson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197683#comment-15197683
]
Philip Thompson commented on CASSANDRA-11279:
-
[~krummas], would writing more data possibly help? Or is the problem the
allocation of vnodes to disks?
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Philip Thompson
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190742#comment-15190742
]
Marcus Eriksson commented on CASSANDRA-11279:
-
I guess it is a bad test - the randomness of vnode allocation makes the disks
unbalanced. I will have a look if I can improve the test
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189539#comment-15189539
]
Jim Witschey commented on CASSANDRA-11279:
--
I'm going to consider the following failure as part of the same bug here:
http://cassci.datastax.com/job/trunk_offheap_dtest/65/testReport/disk_balance_test/TestDiskBalance/disk_balance_decommission_test/
This is a failure on the offheap-memtable trunk job; it's only flapped once,
but is worth looking into. [~krummas], what's expected behavior here?
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176716#comment-15176716
]
Russ Hatch commented on CASSANDRA-11279:
This assert is possibly just a bit too stringent, values above would be about
26% below max and the assertion error wants 20% or less.
> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
