Re: nodetool repair stalled
Hi Eric, The data are stored in JBOD. Only one of the disk got crashed other 3 disk still holds the old data . That's why I didn't clean the whole node and issue a fresh restart Thanks Rob. Will do try that way. From: Eric Stevens Sent: Wednesday, November 12, 2014 8:21 PM To: user@cassandra.apache.org Wouldn't it be a better idea to issue removenode on the crashed node, wipe the whole data directory (including system) and let it bootstrap cleanly so that it's not part of the cluster while it gets back up to speed? On Tue, Nov 11, 2014, 12:32 PM Robert Coli rc...@eventbrite.com wrote: On Tue, Nov 11, 2014 at 10:48 AM, venkat sam samvenkat...@outlook.com wrote: I have a 5 node cluster. In one node one of the data directory partition got crashed. After disk replacement I restarted the Cassandra daemon and gave nodetool repair to restore the missing replica’s. But nodetool repair is getting stuck after syncing one of the columnfamily Yes, nodetool repair often hangs. Search through the archives, but the summary is. 1) try to repair CFs one at a time 2) it's worse with vnodes 3) try tuning the phi detector or network stream timeouts =Rob
Re: The data didn't spread evenly on disks
What compaction strategy are you using? venkat From: Yatong Zhang Sent: Saturday, November 1, 2014 12:32 PM To: user@cassandra.apache.org Hi there, I am using 2.0.10 and my Cassandra node has 6 disks and I configured 6 data directories in cassandra.yaml. But the data was not evenly stored on these 6 disks: disk1 67% used disk2 100% used disk3 100% used disk4 76% used disk5 69% used disk6 81% used So: 1. Is there a way to make the data evenly spread on disks? 2. I set 'disk_failure_policy: best_effort', so when the disk is full, will it serve reading or just stop working at all? 3. Any other suggestions?
RE: Commissioning failure
Thanks Rob. We have disabled firewalls between the nodes and yet getting the same error. I have one more doubt. In our cluster we configured size tiered compaction strategy on JBOD and we are facing issues like uneven distribution of data. Is it possible to change compaction strategy of tables with data? Which compaction strategy will be ideal for data directories in JBOD ? Date: Thu, 30 Oct 2014 11:16:59 -0700 Subject: Re: Commissioning failure From: rc...@eventbrite.com To: user@cassandra.apache.org On Wed, Oct 29, 2014 at 10:39 PM, Aravindan T aravinda...@tcs.com wrote: What could be the reasons for the stream error other than SSTABLE corruption? There's tons of reasons streams fail. Cassandra team is aware of how painful it makes things, so they are working on them. Be sure that a firewall is not dropping long running connections. =Robhttp://twitter.com/rcolidba
RE: Commissioning failure
Yes Hannu, Initially for one month we didn't face any problem. But once tables got bigger we are getting the disk space issues. Can I change the compaction strategy. Will changing compaction strategy have an impact on data consistency? Will it cause corruption of SSTable . -Venkat Date: Fri, 31 Oct 2014 13:43:11 +0200 Subject: Re: Commissioning failure From: hkro...@gmail.com To: user@cassandra.apache.org Hi, I think only LevelledCompactionStrategy makes sense on JBOD because that can distribute data more evenly. Although I don't know what is the exact strategy where each compaction strategy will store sstables. If you use SizeTieredCompactionStrategy you might run into problems when sstables get big enough. I think the theoretical maximum size for an STCS table is the half of one disk size. Because you might be able to run major compaction in that situation. I wouldn't actually use JBOD with anything else than LCS. With smaller tables it doesn't really matter that much but with big tables you would run into problem with anything else. Please correct me if I am wrong. Hannu 2014-10-31 12:16 GMT+02:00 venkat sam samvenkat...@outlook.com: Thanks Rob. We have disabled firewalls between the nodes and yet getting the same error. I have one more doubt. In our cluster we configured size tiered compaction strategy on JBOD and we are facing issues like uneven distribution of data. Is it possible to change compaction strategy of tables with data? Which compaction strategy will be ideal for data directories in JBOD ? Date: Thu, 30 Oct 2014 11:16:59 -0700 Subject: Re: Commissioning failure From: rc...@eventbrite.com To: user@cassandra.apache.org On Wed, Oct 29, 2014 at 10:39 PM, Aravindan T aravinda...@tcs.com wrote: What could be the reasons for the stream error other than SSTABLE corruption? There's tons of reasons streams fail. Cassandra team is aware of how painful it makes things, so they are working on them. Be sure that a firewall is not dropping long running connections. =Robhttp://twitter.com/rcolidba
Commissioning failure
I have a 5 node cassandra cluster and i commissioned 1 new node to the cluster. when i added 1 node. it received streams from 3 nodes out of which 2 were completed successfully and one stream got failed. how can i resume the stream which has failed?
