[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153283#comment-16153283
]
Marcus Eriksson commented on CASSANDRA-13738:
-
failing dtests pass locally
committed as {{4e834c53ca57910e8c4}}, thanks!
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150783#comment-16150783
]
Jay Zhuang commented on CASSANDRA-13738:
2.2 branch uTest fail for {{ant eclipse-warnings}}, but I'm unable to reproduce
it locally:
{noformat}
eclipse-warnings:
[mkdir] Created dir: /home/ubuntu/cassandra/build/ecj
[echo] Running Eclipse Code Analysis. Output logged to
/home/ubuntu/cassandra/build/ecj/eclipse_compiler_checks.txt
[java] incorrect classpath: /home/ubuntu/cassandra/build/cobertura/classes
[java] --
[java] 1. ERROR in
/home/ubuntu/cassandra/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
(at line 853)
[java] ISSTableScanner scanner = cleanupStrategy.getScanner(sstable,
getRateLimiter());
[java] ^^^
[java] Resource 'scanner' should be managed by try-with-resource
[java] --
[java] --
[java] 2. ERROR in
/home/ubuntu/cassandra/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
(at line 257)
[java] scanners.add(new LeveledScanner(intersecting, range));
[java] ^^^
[java] Potential resource leak: '' may not be
closed
[java] --
[java] --
[java] 3. ERROR in
/home/ubuntu/cassandra/src/java/org/apache/cassandra/tools/SSTableExport.java
(at line 315)
[java] ISSTableScanner scanner = reader.getScanner();
[java] ^^^
[java] Resource 'scanner' should be managed by try-with-resource
[java] --
[java] 3 problems (3 errors)
{noformat}
And for the other test failures, I don't think they're introduced by this patch.
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149989#comment-16149989
]
Jay Zhuang commented on CASSANDRA-13738:
Yeah, all the builds are failing after the parallelism is set to 4 :(, rebased
the code and updated the unittest:
| branch | utest |
| [13738-2.2|https://github.com/cooldoger/cassandra/tree/13738-2.2] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/13738-2.2.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13738-2.2]
|
| [13738-3.0|https://github.com/cooldoger/cassandra/tree/13738-3.0] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/13738-3.0.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13738-3.0]
|
| [13738-3.11|https://github.com/cooldoger/cassandra/tree/13738-3.11] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/13738-3.11.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13738-3.11]
|
| [13738-trunk|https://github.com/cooldoger/cassandra/tree/13738-trunk] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/13738-trunk.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/13738-trunk]
|
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147621#comment-16147621
]
Jay Zhuang commented on CASSANDRA-13738:
[~iamaleksey] Thanks for the reminder, updated setting and rerunning the tests.
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147439#comment-16147439
]
Aleksey Yeschenko commented on CASSANDRA-13738:
---
[~jay.zhuang] You aren't running all the unit tests, FYI - because there is no
way to get a green run currently. You have parallelism set to 1 instead of 4,
which skips long-test, test-compression, and stress-test. Should set it to 4
and rerun.
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146852#comment-16146852
]
Marcus Eriksson commented on CASSANDRA-13738:
-
lgtm
running dtests just to be sure:
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/237/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/238/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/239/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/240/
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145189#comment-16145189
]
Aleksey Yeschenko commented on CASSANDRA-13738:
---
[~jjirsa] Not my strongest area of the codebase. Maybe [~krummas] has some
spare cycles?
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143212#comment-16143212
]
Jeff Jirsa commented on CASSANDRA-13738:
[~iamaleksey] - are you willing to take review on this?
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115899#comment-16115899
]
Jay Zhuang commented on CASSANDRA-13738:
Updated unittest to make it stable:
| branch | utest |
| [13738-2.2|https://github.com/cooldoger/cassandra/tree/13738-2.2] |
[circleci#70 passed|https://circleci.com/gh/cooldoger/cassandra/70] |
| [13738-3.0|https://github.com/cooldoger/cassandra/tree/13738-3.0] |
[circleci#69 passed|https://circleci.com/gh/cooldoger/cassandra/69] |
| [13738-3.11|https://github.com/cooldoger/cassandra/tree/13738-3.11] |
[circleci#68 passed|https://circleci.com/gh/cooldoger/cassandra/68] |
| [13738-trunk|https://github.com/cooldoger/cassandra/tree/trunk] |
[circleci#67 passed|https://circleci.com/gh/cooldoger/cassandra/67] |
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113782#comment-16113782
]
Jeff Jirsa commented on CASSANDRA-13738:
I'll take review but I'm at least two weeks away from getting to it
If anyone else beats me to it I won't mind
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113737#comment-16113737
]
Jay Zhuang commented on CASSANDRA-13738:
I deployed the change to the cluster which has the problem. Confirmed the issue
has been fixed. It has been running for more than 6 hours, so far looks fine.
[~jjirsa] do you mind reviewing the patch?
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112230#comment-16112230
]
Jay Zhuang commented on CASSANDRA-13738:
I think the problem has been there for awhile. It only happens when [the
IndexSummary is
rebuilt|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryRedistribution.java#L129],
which is triggered by [large read traffic load
changing|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryRedistribution.java#L202].
I'm able to reproduce it with an unittest. Here is the patch, please review:
| branch | utest |
| [13738-2.2|https://github.com/cooldoger/cassandra/tree/13738-2.2] |
[circleci#53|https://circleci.com/gh/cooldoger/cassandra/53] |
| [13738-3.0|https://github.com/cooldoger/cassandra/tree/13738-3.0] |
[circleci#52|https://circleci.com/gh/cooldoger/cassandra/52] |
| [13738-3.11|https://github.com/cooldoger/cassandra/tree/13738-3.11] |
[circleci#51|https://circleci.com/gh/cooldoger/cassandra/51] |
| [13738-trunk|https://github.com/cooldoger/cassandra/tree/trunk] |
[circleci#50|https://circleci.com/gh/cooldoger/cassandra/50] |
Seems branch {{2.1}} don't have this issue.
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112126#comment-16112126
]
Kurt Greaves commented on CASSANDRA-13738:
--
This has been around for a long time - haven't had the opportunity to find out
what the exact cause was but this makes sense. I've definitely seen it in 3.7.
Pretty sure I've also seen it in 3.0 and 2.1 as well. I don't think it happens
in all versions, or at least for some reason it doesn't happen on all clusters.
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112060#comment-16112060
]
Jeff Jirsa commented on CASSANDRA-13738:
Are you aware yet if this is a new regression? If so, when was it introduced?
> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but
> {{nodetool status}} shows far more load than it actually is and keeps
> increasing, restarting the process will reset the load, but keeps increasing
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens Owns (effective) Host ID
> Rack
> UN IP1* 13.52 TB 256 100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d rac1
> UN IP2* 14.25 TB 256 100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1
> UN IP3* 13.52 TB 256 100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1
> UN IP4* 22.13 TB 256 100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1
> UN IP5* 18.02 TB 256 100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1
> UN IP6* 11.68 TB 256 100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically
> executes hourly), the updated SSTable size is added again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
