Re: Increased replication factor not evident in CLI
Just to completely eliminate the possibility of the same bug, if you look here: http://www.mail-archive.com/dev@cassandra.apache.org/msg04992.html If you create a test keyspace, and look at the timestamp in the schema_keyspaces column family in comparison to your existing keyspace, is that timestamp greater? Thanks, -Mike On Jul 12, 2012, at 8:56 PM, Michael Theroux wrote: Sounds a lot like a bug that I hit that was filed and fixed recently: https://issues.apache.org/jira/browse/CASSANDRA-4432 -Mike On Jul 12, 2012, at 8:16 PM, Edward Capriolo wrote: Possibly the bug with nanotime causing cassandra to think the change happened in the past. Talked about onlist in past few days. On Thursday, July 12, 2012, aaron morton aa...@thelastpickle.com wrote: Do multiple nodes say the RF is 2 ? Can you show the output from the CLI ? Do show schema and show keyspace say the same thing ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/07/2012, at 7:39 AM, Dustin Wenz wrote: We recently increased the replication factor of a keyspace in our cassandra 1.1.1 cluster from 2 to 4. This was done by setting the replication factor to 4 in cassandra-cli, and then running a repair on each node. Everything seems to have worked; the commands completed successfully and disk usage increased significantly. However, if I perform a describe on the keyspace, it still shows replication_factor:2. So, it appears that the replication factor might be 4, but it reports as 2. I'm not entirely sure how to confirm one or the other. Since then, I've stopped and restarted the cluster, and even ran an upgradesstables on each node. The replication factor still doesn't report as I would expect. Am I missing something here? - .Dustin
Re: Increased replication factor not evident in CLI
It sounds plausible that is what we are running into. All of our nodes report a replication factor of 2 (both using describe, and show schema), even though the cluster reported that all schemas agree after I issued the change to 4. If this is related to the bug that you filed, it might also explain why I've had difficulty changing the compression options on this same cluster. I issue an update command, schemas agree, but yet the change is not evident. - .Dustin On Jul 12, 2012, at 7:56 PM, Michael Theroux wrote: Sounds a lot like a bug that I hit that was filed and fixed recently: https://issues.apache.org/jira/browse/CASSANDRA-4432 -Mike On Jul 12, 2012, at 8:16 PM, Edward Capriolo wrote: Possibly the bug with nanotime causing cassandra to think the change happened in the past. Talked about onlist in past few days. On Thursday, July 12, 2012, aaron morton aa...@thelastpickle.com wrote: Do multiple nodes say the RF is 2 ? Can you show the output from the CLI ? Do show schema and show keyspace say the same thing ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/07/2012, at 7:39 AM, Dustin Wenz wrote: We recently increased the replication factor of a keyspace in our cassandra 1.1.1 cluster from 2 to 4. This was done by setting the replication factor to 4 in cassandra-cli, and then running a repair on each node. Everything seems to have worked; the commands completed successfully and disk usage increased significantly. However, if I perform a describe on the keyspace, it still shows replication_factor:2. So, it appears that the replication factor might be 4, but it reports as 2. I'm not entirely sure how to confirm one or the other. Since then, I've stopped and restarted the cluster, and even ran an upgradesstables on each node. The replication factor still doesn't report as I would expect. Am I missing something here? - .Dustin
Re: Increased replication factor not evident in CLI
I was able to apply the patch in the cited bug report to the public source for version 1.1.2. It seemed pretty straightforward; six lines in MigrationManager.java were switched from System.currentTimeMillis() to FBUtilities.timestampMicros(). I then re-built the project by running 'ant artifacts' in the cassandra root. After I was up and running with the new version, I attempted to increase the replication factor, and then the compressions options. Unfortunately, new patch did not seem to help in my case. Neither of the schema attributes would change. Running a describe cluster shows that all node schemas are consistent. Are there any other ways that I could potentially force Cassandra to accept these changes? - .Dustin On Jul 13, 2012, at 10:02 AM, Dustin Wenz wrote: It sounds plausible that is what we are running into. All of our nodes report a replication factor of 2 (both using describe, and show schema), even though the cluster reported that all schemas agree after I issued the change to 4. If this is related to the bug that you filed, it might also explain why I've had difficulty changing the compression options on this same cluster. I issue an update command, schemas agree, but yet the change is not evident. - .Dustin On Jul 12, 2012, at 7:56 PM, Michael Theroux wrote: Sounds a lot like a bug that I hit that was filed and fixed recently: https://issues.apache.org/jira/browse/CASSANDRA-4432 -Mike On Jul 12, 2012, at 8:16 PM, Edward Capriolo wrote: Possibly the bug with nanotime causing cassandra to think the change happened in the past. Talked about onlist in past few days. On Thursday, July 12, 2012, aaron morton aa...@thelastpickle.com wrote: Do multiple nodes say the RF is 2 ? Can you show the output from the CLI ? Do show schema and show keyspace say the same thing ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/07/2012, at 7:39 AM, Dustin Wenz wrote: We recently increased the replication factor of a keyspace in our cassandra 1.1.1 cluster from 2 to 4. This was done by setting the replication factor to 4 in cassandra-cli, and then running a repair on each node. Everything seems to have worked; the commands completed successfully and disk usage increased significantly. However, if I perform a describe on the keyspace, it still shows replication_factor:2. So, it appears that the replication factor might be 4, but it reports as 2. I'm not entirely sure how to confirm one or the other. Since then, I've stopped and restarted the cluster, and even ran an upgradesstables on each node. The replication factor still doesn't report as I would expect. Am I missing something here? - .Dustin
Increased replication factor not evident in CLI
We recently increased the replication factor of a keyspace in our cassandra 1.1.1 cluster from 2 to 4. This was done by setting the replication factor to 4 in cassandra-cli, and then running a repair on each node. Everything seems to have worked; the commands completed successfully and disk usage increased significantly. However, if I perform a describe on the keyspace, it still shows replication_factor:2. So, it appears that the replication factor might be 4, but it reports as 2. I'm not entirely sure how to confirm one or the other. Since then, I've stopped and restarted the cluster, and even ran an upgradesstables on each node. The replication factor still doesn't report as I would expect. Am I missing something here? - .Dustin
Re: Increased replication factor not evident in CLI
Do multiple nodes say the RF is 2 ? Can you show the output from the CLI ? Do show schema and show keyspace say the same thing ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/07/2012, at 7:39 AM, Dustin Wenz wrote: We recently increased the replication factor of a keyspace in our cassandra 1.1.1 cluster from 2 to 4. This was done by setting the replication factor to 4 in cassandra-cli, and then running a repair on each node. Everything seems to have worked; the commands completed successfully and disk usage increased significantly. However, if I perform a describe on the keyspace, it still shows replication_factor:2. So, it appears that the replication factor might be 4, but it reports as 2. I'm not entirely sure how to confirm one or the other. Since then, I've stopped and restarted the cluster, and even ran an upgradesstables on each node. The replication factor still doesn't report as I would expect. Am I missing something here? - .Dustin
Re: Increased replication factor not evident in CLI
Possibly the bug with nanotime causing cassandra to think the change happened in the past. Talked about onlist in past few days. On Thursday, July 12, 2012, aaron morton aa...@thelastpickle.com wrote: Do multiple nodes say the RF is 2 ? Can you show the output from the CLI ? Do show schema and show keyspace say the same thing ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/07/2012, at 7:39 AM, Dustin Wenz wrote: We recently increased the replication factor of a keyspace in our cassandra 1.1.1 cluster from 2 to 4. This was done by setting the replication factor to 4 in cassandra-cli, and then running a repair on each node. Everything seems to have worked; the commands completed successfully and disk usage increased significantly. However, if I perform a describe on the keyspace, it still shows replication_factor:2. So, it appears that the replication factor might be 4, but it reports as 2. I'm not entirely sure how to confirm one or the other. Since then, I've stopped and restarted the cluster, and even ran an upgradesstables on each node. The replication factor still doesn't report as I would expect. Am I missing something here? - .Dustin
Re: Increased replication factor not evident in CLI
Sounds a lot like a bug that I hit that was filed and fixed recently: https://issues.apache.org/jira/browse/CASSANDRA-4432 -Mike On Jul 12, 2012, at 8:16 PM, Edward Capriolo wrote: Possibly the bug with nanotime causing cassandra to think the change happened in the past. Talked about onlist in past few days. On Thursday, July 12, 2012, aaron morton aa...@thelastpickle.com wrote: Do multiple nodes say the RF is 2 ? Can you show the output from the CLI ? Do show schema and show keyspace say the same thing ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/07/2012, at 7:39 AM, Dustin Wenz wrote: We recently increased the replication factor of a keyspace in our cassandra 1.1.1 cluster from 2 to 4. This was done by setting the replication factor to 4 in cassandra-cli, and then running a repair on each node. Everything seems to have worked; the commands completed successfully and disk usage increased significantly. However, if I perform a describe on the keyspace, it still shows replication_factor:2. So, it appears that the replication factor might be 4, but it reports as 2. I'm not entirely sure how to confirm one or the other. Since then, I've stopped and restarted the cluster, and even ran an upgradesstables on each node. The replication factor still doesn't report as I would expect. Am I missing something here? - .Dustin
