Re: Repair with –pr stuck in between on Cassandra 3.11.1
Hi Paulo, No we are not using JBOD.Just a bunch of disks. Thanks On Thu, 25 Jan 2018 at 5:44 PM, Paulo Motta wrote: > Are you using JBOD? A thread dump (jstack ) on the affected nodes > would probably help troubleshoot this. > > 2018-01-25 6:45 GMT-02:00 shini gupta : > > Hi, > > > > > > We have upgraded the system from Cassandra 2.1.16 to 3.11.1. After about > > 335M of data loading, repair with –pr and –full option was triggered for > > all keyspaces (7 keyspaces). Data size of main_k1 is ~170G. > > > > > > > > Please find the command executed for repair: > > > > > > > > …./cassandra/bin/nodetool -u - -p 7199 repair -full > –tr > > > > > > > > It was observed that the repair for the first keyspace i.e main_k1 got > > finished in 2 hrs at 13:39 hrs. Even after about 1:30 hrs, there were > just > > warning messages about the orphaned sstables. The repair command for the > > first keyspace completed successfully but then the nodetool repair > command > > got stuck and there was no output for remaining 6 keyspaces. > > > > > > > > Please find the system logs below: > > > > > > > > INFO [Repair-Task-2] 2018-01-18 11:35:01,161 RepairRunnable.java:139 - > > Starting repair command #1 (84dbf590-fc15-11e7-85c6-1594c4c73c8e), > repairing > > keyspace main_k1 with repair options (parallelism: parallel, primary > range: > > true, incremental: false, job threads: 1, ColumnFamilies: [], > dataCenters: > > [], hosts: [], # of ranges: 256, pull repair: false) > > > > > > > > INFO [CompactionExecutor:39] 2018-01-18 13:39:58,174 > > RepairRunnable.java:343 - Repair command #1 finished in 2 hours 4 > minutes 57 > > seconds > > > > > > > > WARN [ValidationExecutor:12] 2018-01-18 14:30:49,356 > > LeveledCompactionStrategy.java:273 - Live sstable > > > …../data/main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22184-big-Data.db > > from level 3 is not on corresponding level in the leveled manifest. This > is > > not a problem per se, but may indicate an orphaned sstable due to a > failed > > compaction not cleaned up properly. > > > > . > > > > > > > > WARN [ValidationExecutor:26] 2018-01-18 15:03:53,598 > > LeveledCompactionStrategy.java:273 - Live sstable …../data > > /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22291-big-Data.db > from > > level 2 is not on corresponding level in the leveled manifest. This is > not a > > problem per se, but may indicate an orphaned sstable due to a failed > > compaction not cleaned up properly. > > > > WARN [ValidationExecutor:26] 2018-01-18 15:03:53,598 > > LeveledCompactionStrategy.java:273 - Live sstable …../data > > /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22216-big-Data.db > from > > level 3 is not on corresponding level in the leveled manifest. This is > not a > > problem per se, but may indicate an orphaned sstable due to a failed > > compaction not cleaned up properly. > > > > > > > > Is anybody facing such issues with repair –pr for cassandra 3.11.1 ? Is > this > > behavior due to the warning messages reported in logs. > > > > > > > > Thanks. > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- -Shini Gupta ""Trusting in God won't make the mountain smaller, But will make climbing easier. Do not ask God for a lighter load But ask Him for a stronger back... ""
Repair with –pr stuck in between on Cassandra 3.11.1
Hi, We have upgraded the system from Cassandra 2.1.16 to 3.11.1. After about 335M of data loading, repair with –pr and –full option was triggered for all keyspaces (7 keyspaces). Data size of main_k1 is ~170G. Please find the command executed for repair: *…./cassandra/bin/nodetool -u - -p 7199 repair -full –tr* It was observed that the repair for the first keyspace i.e main_k1 got finished in 2 hrs at 13:39 hrs. Even after about 1:30 hrs, there were just warning messages about the orphaned sstables. The repair command for the first keyspace completed successfully but then the nodetool repair command got stuck and there was no output for remaining 6 keyspaces. Please find the system logs below: *INFO [Repair-Task-2] 2018-01-18 11:35:01,161 RepairRunnable.java:139 - Starting repair command #1 (84dbf590-fc15-11e7-85c6-1594c4c73c8e), repairing keyspace main_k1 with repair options (parallelism: parallel, primary range: true, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 256, pull repair: false)* *INFO [CompactionExecutor:39] 2018-01-18 13:39:58,174 RepairRunnable.java:343 - Repair command #1 finished in 2 hours 4 minutes 57 seconds* *WARN [ValidationExecutor:12] 2018-01-18 14:30:49,356 LeveledCompactionStrategy.java:273 - Live sstable ** …..**/data/main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22184-big-Data.db from level 3 is not on corresponding level in the leveled manifest. This is not a problem per se, but may indicate an orphaned sstable due to a failed compaction not cleaned up properly.* *.* *WARN [ValidationExecutor:26] 2018-01-18 15:03:53,598 LeveledCompactionStrategy.java:273 - Live sstable **…..**/data /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22291-big-Data.db from level 2 is not on corresponding level in the leveled manifest. This is not a problem per se, but may indicate an orphaned sstable due to a failed compaction not cleaned up properly.* *WARN [ValidationExecutor:26] 2018-01-18 15:03:53,598 LeveledCompactionStrategy.java:273 - Live sstable **…..**/data /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22216-big-Data.db from level 3 is not on corresponding level in the leveled manifest. This is not a problem per se, but may indicate an orphaned sstable due to a failed compaction not cleaned up properly.* Is anybody facing such issues with repair –pr for cassandra 3.11.1 ? Is this behavior due to the warning messages reported in logs. Thanks.
Re: 3.0.15 or 3.11.1
Hi, Can you please provide dome JIRAs for superior fixes and performance improvements which are present in 3.11.1 but are missing in 3.0.15. In addition,could you also provide the probable implications of the open memory leak issue in Cassandra 3.11. CASSANDRA-13929: BTree$Builder / io.netty.util.Recycler$Stack leaking memory <https://issues.apache.org/jira/browse/CASSANDRA-13929> Is it still recommended to go for 3.11. Thanks On Mon, Jan 8, 2018 at 2:31 AM, Jon Haddad wrote: > There’s a tweak to TWCS in 3.11.1 that lets data expire faster, but I > wouldn’t call it unstable in any version I’ve ever used it with. I’ve > deployed it on 2.0, 2.1, 2.2 [1], and used it in every version of C* that > we’ve shipped it, and have never had an issue. > > I would put 3.11.1 in prod over 3.0, there’s a number of performance > improvements and a few nice features that make it worth it. Off the top of > my head, off heap memtables, a nice LIMIT optimization, and more flexible > allow filtering options are all nice. > > [1] http://thelastpickle.com/blog/2017/01/10/twcs-part2.html > > > > On Jan 7, 2018, at 2:33 AM, shalom sagges wrote: > > Thanks Guys! > > Sorry for the late reply. > I'm interested in TWCS where I understand is more stable in 3.11.1 than in > 3.0.15, tombstone compaction and slow logs. > > I don't plan to use MVs and SASI in the near future, as I understand are > not Production ready. > > Is it okay to use the above features? > > > > > > On Thu, Jan 4, 2018 at 1:07 AM, Mick Semb Wever > wrote: > >> >> >>> I want to upgrade from 2.x to 3.x. >>> >>> I can definitely use the features in 3.11.1 but it's not a must. >>> So my question is, is 3.11.1 stable and suitable for Production compared >>> to 3.0.15? >>> >> >> >> Use 3.11.1 and don't use any 3.0.x or 3.x features. >> 3.11.1 is effectively three sequential patch releases, and the tick-tock >> releases offered a number of superior fixes and performance improvements >> over what was done in 3.0.x. >> >> Introduce the use of new features later on, one at a time, after thorough >> testing and staging. >> >> regards, >> Mick >> > > > -- -Shini Gupta ""Trusting in God won't make the mountain smaller, But will make climbing easier. Do not ask God for a lighter load But ask Him for a stronger back... ""
Re: Running repair while Cassandra upgrade 2.0.X to 2.1.X
Thanks Alain !! During Cassandra upgrade, we halt repair scheduling for 24 hours as this is good time to finish upgradesstables..we dont want continuous manual monitoring of upgradesstables and then rescheduling of repair tasks when upgradesstables successfully finishes on all the nodes..Here is the problem with the approach..sometimes we see that upgradsstables has finished on a node but still some sstable files exist in the old format. I want to understand how safe it is if our repair executes after 24 hours when old and new sstables cooexist on some nodes. My understanding of https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-5772 says that this is supported and nothing could go wrong ..Can someone please confirm my understanding of the JIRA and confirm that repairs cant break anything in such scenarios?" Thanks Shini On Fri, 8 Dec 2017 at 2:26 PM, Alain RODRIGUEZ wrote: > Hi Shini, > > First thing that comes to my mind honestly is "why?". Why would you do > this? It's way safer to finish upgrade then repair I would say. > > That being said, I can make guesses for you here, but the best would > probably be to test it. > > 1. Running nodetool repair on one of the nodes while upgradesstables is >> still executing on one or more nodes in the cluster. > > 2. Running nodetool repair when upgradesstables failed abruptly on some of >> the nodes such that some sstable files are in new format while other >> sstable files are still in old format. > > > These impacts are unknown for me since I never did this, I always heard > around it was really bad. For me both case are similar. You are going to > repair data with distinct SSTable formats. I don't remember the differences > between Cassandra 2.0 and 2.1. > > In best case it will work. > > But my guess is it might lead to: > - Schema disagreement (but I heard that on schema changes on multi-version > clusters). I don't think you would hit this one. > - Mixing all new and old over the cluster (even on nodes with upgraded > SSTables)... > - Maybe directly fail if networking changed between the 2 versions. > > I would stay away from repairs and upgrade first all the nodes. > > If that's really not doable, then I recommend you to test it (ccm / AWS / > stage cluster, as you see fit). > > Sorry I cannot be more precise, impacts are not clear to me because I > always stayed away from this kind of mixed operations. The need for > repairing before upgrading sstables is not clear to me either. I would add > that I found out in the past that working in a rush on Cassandra often > creates more issues than solutions in general. > > C*heers, > --- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2017-12-08 5:26 GMT+00:00 shini gupta : > >> Hi >> Can someone please answer this query? >> >> Thanks >> >> On Wed, Dec 6, 2017 at 9:58 AM, shini gupta >> wrote: >> >>> If we have upgraded Cassandra binaries from 2.0 to 2.1 on ALL the nodes >>> but upgradesstable is still pending, please provide the impact of following >>> scenarios: >>> >>> >>> >>> 1. Running nodetool repair on one of the nodes while upgradesstables is >>> still executing on one or more nodes in the cluster. >>> >>> 2. Running nodetool repair when upgradesstables failed abruptly on some >>> of the nodes such that some sstable files are in new format while other >>> sstable files are still in old format. >>> >>> >>> >>> Even though it may not be recommended to run I/O intensive operations >>> like repair and upgradesstables simultaneously, can we assume that both the >>> above sceanrios are now supported and will not break anything, especially >>> after https://issues.apache.org/jira/browse/CASSANDRA-5772 has been >>> fixed in 2.0? >>> >>> >>> Regards >>> Shini >>> >>> >> >> >> -- >> -Shini Gupta >> >> ""Trusting in God won't make the mountain smaller, >> But will make climbing easier. >> Do not ask God for a lighter load >> But ask Him for a stronger back... "" >> > > -- -Shini Gupta ""Trusting in God won't make the mountain smaller, But will make climbing easier. Do not ask God for a lighter load But ask Him for a stronger back... ""
Re: Running repair while Cassandra upgrade 2.0.X to 2.1.X
Hi Can someone please answer this query? Thanks On Wed, Dec 6, 2017 at 9:58 AM, shini gupta wrote: > If we have upgraded Cassandra binaries from 2.0 to 2.1 on ALL the nodes > but upgradesstable is still pending, please provide the impact of following > scenarios: > > > > 1. Running nodetool repair on one of the nodes while upgradesstables is > still executing on one or more nodes in the cluster. > > 2. Running nodetool repair when upgradesstables failed abruptly on some of > the nodes such that some sstable files are in new format while other > sstable files are still in old format. > > > > Even though it may not be recommended to run I/O intensive operations like > repair and upgradesstables simultaneously, can we assume that both the > above sceanrios are now supported and will not break anything, especially > after https://issues.apache.org/jira/browse/CASSANDRA-5772 has been fixed > in 2.0? > > > Regards > Shini > > -- -Shini Gupta ""Trusting in God won't make the mountain smaller, But will make climbing easier. Do not ask God for a lighter load But ask Him for a stronger back... ""
Running repair while Cassandra upgrade 2.0.X to 2.1.X
If we have upgraded Cassandra binaries from 2.0 to 2.1 on ALL the nodes but upgradesstable is still pending, please provide the impact of following scenarios: 1. Running nodetool repair on one of the nodes while upgradesstables is still executing on one or more nodes in the cluster. 2. Running nodetool repair when upgradesstables failed abruptly on some of the nodes such that some sstable files are in new format while other sstable files are still in old format. Even though it may not be recommended to run I/O intensive operations like repair and upgradesstables simultaneously, can we assume that both the above sceanrios are now supported and will not break anything, especially after https://issues.apache.org/jira/browse/CASSANDRA-5772 has been fixed in 2.0? Regards Shini
Fwd: Stable Cassandra 3.x version for production
Hi Which version of Cassandra 3.x is stable and production-ready? Regards
