[
https://issues.apache.org/jira/browse/CASSANDRA-10430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
julia zhang updated CASSANDRA-10430:
------------------------------------
Attachment: system.log.4.zip
system.log.3.zip
system.log.2.zip
Attached system log files from a node, whose "Load" became 1.1TB after running
incremental repair
> "Load" report from "nodetool status" is inaccurate
> --------------------------------------------------
>
> Key: CASSANDRA-10430
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10430
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: Cassandra v2.1.9 running on 6 node Amazon AWS, vnodes
> enabled.
> Reporter: julia zhang
> Fix For: 2.1.x
>
> Attachments: system.log.2.zip, system.log.3.zip, system.log.4.zip
>
>
> After running an incremental repair, nodetool status report unbalanced load
> among cluster.
> $ nodetool status mykeyspace
> ==========================
> ||Status|| Address ||Load ||Tokens ||Owns (effective)
> ||Host ID || Rack ||
> |UN |10.1.1.1 |1.13 TB |256 |48.5%
> |a4477534-a5c6-4e3e-9108-17a69aebcfc0| RAC1|
> |UN |10.1.1.2 |2.58 TB |256 |50.5%
> |1a7c3864-879f-48c5-8dde-bc00cf4b23e6 |RAC2|
> |UN |10.1.1.3 |1.49 TB |256 |51.5%
> |27df5b30-a5fc-44a5-9a2c-1cd65e1ba3f7 |RAC1|
> |UN |10.1.1.4 |250.97 GB |256 |51.9%
> |9898a278-2fe6-4da2-b6dc-392e5fda51e6 |RAC3|
> |UN |10.1.1.5 |1.88 TB |256 |49.5%
> |04aa9ce1-c1c3-4886-8d72-270b024b49b9 |RAC2|
> |UN |10.1.1.6 |1.3 TB |256 |48.1%
> |6d5d48e6-d188-4f88-808d-dcdbb39fdca5 |RAC3|
> It seems that only 10.1.1.4 reports correct "Load". There is no hints in the
> cluster and report remains the same after running "nodetool cleanup" on each
> node. "nodetool cfstats" shows number of keys are evenly distributed and
> Cassandra data physical disk on each node report about the same usage.
> "nodetool status" report these inaccurate large storage load until we restart
> each node, after the restart, "Load" report match what we've seen from disk.
> We did not see this behavior until upgrade to v2.1.9
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
