Re: Disk configuration in new cluster node
On Tue, Sep 18, 2012 at 1:54 AM, aaron morton aa...@thelastpickle.comwrote: each with several disks having large capacity, totaling 10 - 12 TB. Is this (another) bad idea? Yes. Very bad. If you had 6TB on average system with spinning disks you would measure duration of repairs and compactions in days. If you want to store 12 TB of data you will need more machines. Would it help if I partitioned the computing resources of my physical machines into VMs? For example, I put four VMs on each of three virtual machines, each with a dedicated 2TB drive. I can now have four tokens in the ring and a RF of 3. And of course, I can arrange them into a way that makes the most sense. Is this getting any better, or am I missing the point? Casey
Re: Disk configuration in new cluster node
On Mon, Sep 17, 2012 at 1:19 AM, aaron morton aa...@thelastpickle.comwrote: 4 drives for data and 1 drive for commitlog, How are you configuring the drives ? It's normally best to present one big data volume, e.g. using raid 0, and put the commit log on say the system mirror. Given the advice to use a single RAID 0 volume, I think that's what I'll do. By system mirror, you are referring to the volume on which the OS is installed? Should the volume with the commit log also have multiple disks in a RAID 0 volume? Alternatively, would a RAID 1 setup be reasonable for the system volume/OS, so the system itself can be resilient to disk failure, or would that kill commit performance? Any preference to hardware RAID 0 vs. using something like mdadm? A word of warning. If you put more than 300GB to 400GB per node you may end experience some issues such as repair, compaction or disaster recovery taking a long time. These are simply soft limits that provide a good rule of thumb for HDD based systems with 1 GigE networking. Hmm. My hope was to be able to run a minimal number of nodes and maximize their capacity because it doesn't make sense in my case to build or maintain a large cluster. I wanted to run a two-node setup (RF=1, RCL=ONE, WCL=ALL), each with several disks having large capacity, totaling 10 - 12 TB. Is this (another) bad idea? Casey
Disk configuration in new cluster node
I'm building a new cluster (to replace the broken setup I've written about in previous posts) that will consist of only two nodes. I understand that I'll be sacrificing high availability of writes if one of the nodes goes down, and I'm okay with that. I'm more interested in maintaining high consistency and high read availability. So I've decided to use a write-level consistency of ALL and read-level consistency of ONE. My first question is about the drives in this setup. If I initially set up the system with, say, 4 drives for data and 1 drive for commitlog, and later I decide to add more capacity to the node by adding more drives for data (adding the new data directory entries in cassandra.yaml), will the node balance out the load on the drives, or is it agnostic to usage of drives underlying data directories? My second question has to do with RAID striping. Would it be more useful to stripe the disk with the commitlog or the disks with the data? Of course, with a single striped volume for data directories, it would be more difficult to add capacity to the node later, as I've suggested above. Casey
adding node to cluster
All, I'm adding a new node to an existing cluster that uses ByteOrderedPartitioner. The documentation says that if I don't configure a token, then one will be automatically generated to take load from an existing node. What I'm finding is that when I add a new node, (super) column lookups begin failing (not sure if it was the row lookup failing or the supercolumn lookup failing), and I'm not sure why. I assumed that while the existing node is transitioning data to the new node the affected rows and (super) columns would still be found in the right place. Any idea why these lookups might be failing? When I decommissioned the the new node, the lookups began working again. Any help is appreciated. Regards, Casey
Re: adding node to cluster
On Thu, Aug 30, 2012 at 11:21 AM, Rob Coli rc...@palominodb.com wrote: On Thu, Aug 30, 2012 at 10:18 AM, Casey Deccio ca...@deccio.net wrote: I'm adding a new node to an existing cluster that uses ByteOrderedPartitioner. The documentation says that if I don't configure a token, then one will be automatically generated to take load from an existing node. What I'm finding is that when I add a new node, (super) column lookups begin failing (not sure if it was the row lookup failing or the supercolumn lookup failing), and I'm not sure why. 1) You almost never actually want BOP. 2) You never want Cassandra to pick a token for you. IMO and the opinion of many others, the fact that it does this is a bug. Specify a token with initial_token. 3) You never want to use Supercolumns. The project does not support them but currently has no plan to deprecate them. Use composite row keys. 4) Unless your existing cluster consists of one node, you almost never want to add only a single new node to a cluster. In general you want to double it. In summary, you are Doing It just about as Wrong as possible... but on to your actual question ... ! :) Well, at least I'm consistent :) Thanks for the hints. Unfortunately, when I first brought up my system--with the goal of getting it up quickly--I thought BOP and Supercolumns were the way to go. Plus, the small cluster of nodes I was using was on a hodgepodge of hardware. I've since had a chance to think somewhat about redesigning and rearchitecting, but it seems like there's no easy way to convert it properly. Step one was to migrate everything over to a single dedicated node on reasonable hardware, so I could begin the process, which brought me to the issue I initially posted about. But the problem is that this is a live system, so data loss is an issue I'd like to avoid. In what way are the lookups failing? Is there an exception? No exception--just failing in that the data should be there, but isn't. Casey
cassandra upgrade to 1.1 - migration problem
I recently upgraded from cassandra 1.0.10 to 1.1. Everything worked fine in one environment, but after I upgraded in another, I can't find my keyspace. When I run, e.g., cassandra-cli with 'use KeySpace;' It tells me that the keyspace doesn't exist. In the log I see this: ERROR [MigrationStage:1] 2012-05-15 11:39:48,216 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]java.lang.AssertionError at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441) at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339) at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269) at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) at org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) I can see that the data I would expect still seems to be in the new place (/var/lib/cassandra/data/App/ColFamily/App-DomainName-*) on all nodes. What am I missing? Thanks, Casey
Re: cassandra upgrade to 1.1 - migration problem
Here's something new in the logs: ERROR 12:21:09,418 Exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.RuntimeException: Cannot open /var/lib/cassandra/data/system/Versions/system-Versions-hc-35 because partitioner does not match org.apache.cassandra.dht.ByteOrderedPartitioner at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:164) at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Casey On Tue, May 15, 2012 at 12:08 PM, Casey Deccio ca...@deccio.net wrote: I recently upgraded from cassandra 1.0.10 to 1.1. Everything worked fine in one environment, but after I upgraded in another, I can't find my keyspace. When I run, e.g., cassandra-cli with 'use KeySpace;' It tells me that the keyspace doesn't exist. In the log I see this: ERROR [MigrationStage:1] 2012-05-15 11:39:48,216 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]java.lang.AssertionError at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441) at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339) at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269) at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) at org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) I can see that the data I would expect still seems to be in the new place (/var/lib/cassandra/data/App/ColFamily/App-DomainName-*) on all nodes. What am I missing? Thanks, Casey
Re: cassandra upgrade to 1.1 - migration problem
cassandra.yaml on all nodes had ByteOrderedPartitioner with both the previous version and upgraded version. That being said, when I first started up cassandra after upgrading (with the updated .yaml, including ByteOrderedPartitioner) all nodes in the ring appeared to be up. But the load they carried was minimal (KB, as opposed to GB in the previous version), and the keyspace didn't exist. Then when I attempted to restart the daemon on each to see if it would help, but starting up failed on each with the partition error. Casey On Tue, May 15, 2012 at 12:59 PM, Oleg Dulin oleg.du...@liquidanalytics.com wrote: Did you check cassandra.yaml to make sure partitioner there matches what was in your old cluster ? Regards, Oleg Dulin Please note my new office #: 732-917-0159 On May 15, 2012, at 3:22 PM, Casey Deccio wrote: Here's something new in the logs: ERROR 12:21:09,418 Exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.RuntimeException: Cannot open /var/lib/cassandra/data/system/Versions/system-Versions-hc-35 because partitioner does not match org.apache.cassandra.dht.ByteOrderedPartitioner at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:164) at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Casey On Tue, May 15, 2012 at 12:08 PM, Casey Deccio ca...@deccio.net wrote: I recently upgraded from cassandra 1.0.10 to 1.1. Everything worked fine in one environment, but after I upgraded in another, I can't find my keyspace. When I run, e.g., cassandra-cli with 'use KeySpace;' It tells me that the keyspace doesn't exist. In the log I see this: ERROR [MigrationStage:1] 2012-05-15 11:39:48,216 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]java.lang.AssertionError at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441) at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339) at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269) at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) at org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) I can see that the data I would expect still seems to be in the new place (/var/lib/cassandra/data/App/ColFamily/App-DomainName-*) on all nodes. What am I missing? Thanks, Casey
Re: cassandra upgrade to 1.1 - migration problem
On Tue, May 15, 2012 at 5:41 PM, Dave Brosius dbros...@mebigfatguy.comwrote:
The replication factor for a keyspace is stored in the
system.schema_keyspaces column family.
Since you can't view this with cli as the server won't start, the only way
to look at it, that i know of is to use the
sstable2json tool on the *.db file for that column family...
So for instance on my machine i do
./sstable2json /var/lib/cassandra/data/**system/schema_keyspaces/**
system-schema_keyspaces-ia-1-**Data.db
and get
{
7374726573735f6b73: [[durable_writes,true,**1968197311980145],
[name,stress_ks,**1968197311980145], [strategy_class,org.apache.**
cassandra.locator.**SimpleStrategy,**1968197311980145],
[strategy_options,{\**replication_factor\:\3\},**
1968197311980145]]
It's likely you don't have a entry from replication_factor.
Yep, I checked the system.schema_keyspaces ColumnFamily, and there was no
replication_factor value, as you suspected. But the dev cluster that
worked after upgrade did have that value, so it started up okay.
Apparently pre-1.1 was less picky about its presence.
Theoretically i suppose you could embellish the output, and use
json2sstable to fix it, but I have no experience here, and would get the
blessings of datastax fellas, before proceeding.
Actually, I went ahead and took a chance because I had already completely
offline for several hours and wanted to get things back up. I did what you
suggested and added the replication_factor value to the json returned from
sstable2json and imported it using json2sstable. Fortunately I had the dev
cluster values to use as a basis. I started things up, and it worked like
a champ. Thanks!
Casey
Re: NullPointerException on upgradesstables
On Thu, Mar 1, 2012 at 2:39 AM, aaron morton aa...@thelastpickle.comwrote: I am guessing you are running low on disk space. Can you check and try to free some up ? Okay, I've freed some up and am trying again. Looks like a bug in CompactionTask.execute() see https://issues.apache.org/jira/browse/CASSANDRA-3985 Hope that helps. Yes, thanks. Casey
Re: can't find rows
On Thu, Mar 1, 2012 at 9:33 AM, aaron morton aa...@thelastpickle.comwrote: What RF were you using and had you been running repair regularly ? RF 1 *sigh*. Waiting until I have more/better resources to use RF 1. Hopefully soon. In the mean time... Oddly (to me), when I removed the most recently added node, all my rows re-appeared, but were only up-to-date as of a 10 days ago (a few days before I added the node). None of the supercolumns since then show up. But when I look at the sstable files on the different nodes, I see large files with timestamps in between that date and today's, which makes me think the data is still there. Also, if I re-add the troublesome new node (not having run cleanup), all rows are again inaccessible until I again decommission it. Casey
can't find rows
I recently had to do some shuffling with one of my cassandra nodes because it was running out of disk space. I did a few things in the process, and I'm not sure in the end which caused my problem. First I added a second file path to the data directory in cassandra.yaml. Things still worked fine after this, as far as I could tell. Shortly after this, however, I took down the node and rsync'd the data from both data directories, as well as commitlogs, to an external drive. I then shut down the machine, replaced the hard drives with bigger drives, and re-installed the OS. I re-created the data directories, rsync'd the data and commitlogs back over from the external drive, and started up cassandra, re-adding it to the ring. When it came up, all of my rows were missing for one columnfamily and nearly all my rows were missing for another--or at least that's what it looks like, based on walking the rows. I tried scrubbing each of the nodes. One of them had insufficient disk space (yes, this seems to be a recurring problem) for scrub, so I did upgradesstables instead, and that one is still in progress. So far the scrub/upgradesstables hasn't seemed to help. But in the log messages created during scrub/upgradesstables it shows realistic numbers (i.e., in terms of the rows that existed before this ordeal) created in each new sstable. Also, the loads shown when I run nodetool ring still reflects the numbers with the complete set of rows. That's encouraging, but I can't seem to access these phantom rows. Please help! Thanks, Casey
Re: can't find rows
On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio ca...@deccio.net wrote: I recently had to do some shuffling with one of my cassandra nodes because it was running out of disk space. I did a few things in the process, and I'm not sure in the end which caused my problem. First I added a second file path to the data directory in cassandra.yaml. Things still worked fine after this, as far as I could tell. Shortly after this, however, I took down the node and rsync'd the data from both data directories, as well as commitlogs, to an external drive. I then shut down the machine, replaced the hard drives with bigger drives, and re-installed the OS. I re-created the data directories, rsync'd the data and commitlogs back over from the external drive, and started up cassandra, re-adding it to the ring. When it came up, all of my rows were missing for one columnfamily and nearly all my rows were missing for another--or at least that's what it looks like, based on walking the rows. I tried scrubbing each of the nodes. One of them had insufficient disk space (yes, this seems to be a recurring problem) for scrub, so I did upgradesstables instead, and that one is still in progress. So far the scrub/upgradesstables hasn't seemed to help. But in the log messages created during scrub/upgradesstables it shows realistic numbers (i.e., in terms of the rows that existed before this ordeal) created in each new sstable. Also, the loads shown when I run nodetool ring still reflects the numbers with the complete set of rows. That's encouraging, but I can't seem to access these phantom rows. Please help! I neglected to mention that I'm running cassandra 1.0.7. Thanks, Casey
NullPointerException on upgradesstables
Using cassandra 1.0.7, I got the following, as I was trying to rebuild my sstables: $ nodetool -h localhost upgradesstables Error occured while upgrading the sstables for keyspace MyKeySpace java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:203) at org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:219) at org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:995) at org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:1648) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NullPointerException at java.io.File.init(File.java:222) at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:641) at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:652) at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1888) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:151) at org.apache.cassandra.db.compaction.CompactionManager$4.perform(CompactionManager.java:229) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:182) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Any help? Regards, Casey
Re: can't find rows
On Wed, Feb 29, 2012 at 5:29 AM, Casey Deccio ca...@deccio.net wrote: On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio ca...@deccio.net wrote: I recently had to do some shuffling with one of my cassandra nodes because it was running out of disk space. I did a few things in the process, and I'm not sure in the end which caused my problem. First I added a second file path to the data directory in cassandra.yaml. Things still worked fine after this, as far as I could tell. Shortly after this, however, I took down the node and rsync'd the data from both data directories, as well as commitlogs, to an external drive. I then shut down the machine, replaced the hard drives with bigger drives, and re-installed the OS. I re-created the data directories, rsync'd the data and commitlogs back over from the external drive, and started up cassandra, re-adding it to the ring. When it came up, all of my rows were missing for one columnfamily and nearly all my rows were missing for another--or at least that's what it looks like, based on walking the rows. I tried scrubbing each of the nodes. One of them had insufficient disk space (yes, this seems to be a recurring problem) for scrub, so I did upgradesstables instead, and that one is still in progress. So far the scrub/upgradesstables hasn't seemed to help. But in the log messages created during scrub/upgradesstables it shows realistic numbers (i.e., in terms of the rows that existed before this ordeal) created in each new sstable. Also, the loads shown when I run nodetool ring still reflects the numbers with the complete set of rows. That's encouraging, but I can't seem to access these phantom rows. Please help! I neglected to mention that I'm running cassandra 1.0.7. Apologies for replying to my own post (again), but here's the follow up. I decommissioned the newest of the four nodes in the cluster, which was carrying hardly any load (I'm using ByteOrderedPartitioner), but after I decommissioned, rows were available again, but only as they were from 10 days ago. Supercolumns added after that date weren't around. Casey
Re: node stuck leaving
On Tue, Jul 12, 2011 at 10:10 AM, Brandon Williams dri...@gmail.com wrote: On Mon, Jul 11, 2011 at 11:51 PM, Casey Deccio ca...@deccio.net wrote: java.lang.RuntimeException: Cannot recover SSTable with version f (current version g). You need to scrub before any streaming is performed. Okay, turns out that my disk is full, so I'm insufficient disk space errors when running compact or scrub. So, while I'm working on a getting a more production system with better capacity, what would I do to move x.x.x.3's load to x.x.x.2, given the following distribution: $ nodetool -h localhost ring Address DC RackStatus State Load OwnsToken Token(bytes[de4075d0a474c4a773efa2891c020529]) x.x.x.1 datacenter1 rack1 Up Leaving 150.63 GB 33.33% Token(bytes[10956f12b46304bf70412ad0eac14344]) x.x.x.2 datacenter1 rack1 Up Normal 79.21 GB33.33% Token(bytes[50af14df71eafac7bac60fbc836c6722]) x.x.x.3 datacenter1 rack1 Up Normal 60.74 GB33.33% Token(bytes[de4075d0a474c4a773efa2891c020529]) Thanks, Casey
node stuck leaving
I've got a node that is stuck Leaving the ring. Running nodetool decommission never terminates. It's been in this state for about a week, and the load has not decreased: $ nodetool -h localhost ring Address DC RackStatus State Load OwnsToken Token(bytes[de4075d0a474c4a773efa2891c020529]) x.x.x.1 datacenter1 rack1 Up Leaving 150.63 GB 33.33% Token(bytes[10956f12b46304bf70412ad0eac14344]) x.x.x.2 datacenter1 rack1 Up Normal 79.21 GB33.33% Token(bytes[50af14df71eafac7bac60fbc836c6722]) x.x.x.3 datacenter1 rack1 Up Normal 60.74 GB33.33% Token(bytes[de4075d0a474c4a773efa2891c020529]) Any ideas? Regards, Casey
Re: change node IP address
Hi Aaron, I'm using 7.3 (upgrading to 7.4). Basically I needed to swap the IPs of two of my nodes. Unfortunately, after I did so, neither was accessible in the original ring anymore. I saw the warning message in the logs about the token being reassigned. Maybe I didn't give it enough time, but accessibility faltered, so I reverted the IPs of the nodes, decommissioned them each, swapped their IPs again, then brought them back into the ring one at a time. That seemed to work. I admit that I'm still quite a novice when it comes to cassandra's workings, and since my system was unresponsive for a time, my goal was bringing it back up, and I didn't dive into extensive analysis. Casey On Wed, Mar 23, 2011 at 2:16 AM, aaron morton aa...@thelastpickle.comwrote: Which version are you using ? It looks like using 0.7X (and prob 0.6) versions you can just shutdown the node and bring it back up with the new IP and It Just Works https://issues.apache.org/jira/browse/CASSANDRA-872 I've not done it before, anyone else ? Aaron On 23 Mar 2011, at 07:53, Casey Deccio wrote: What is the process of changing the IP address for a node in a cluster? Casey
Re: Reducing memory footprint
On Sat, Mar 5, 2011 at 7:37 PM, aaron morton aa...@thelastpickle.comwrote: There is some additional memory usage in the JVM beyond that Heap size, in the permanent generation. 900mb sounds like too much for that, but you can check by connecting with JConsole and looking at the memory tab. You can also check the heap size there to see that it's under the value you've set. Thanks for the tip! From JConsole: Heap memory usage: Current 46M; Max 902M Non-Heap memory usage: Current 34M; Max 200MB Both of these seem reasonable and don't reach the (current) 2.1 GB resident usage I am seeing. Check you are using standard disk access (in conf/cassandra.yaml) rather than memory mapped access. However the memory mapped memory is reported as virtual memory, not resident. So I'm just mentioning it to be complete. At the moment, it's set to auto, but it's a 64-bit machine, so I believe it's using memory mapped. The virtual memory usage says that it is 54.6 GB. If you think you've configured things correctly and the JVM is not behaving (which is unlikely) please include some information on the JVM and OS versions and some hard numbers about what the process is using. Debian 6.0 using openjdk-6-jre-lib-6b18-1.8.3-2+squeeze1 Thanks for your help. Casey
Reducing memory footprint
I have a small ring of cassandra nodes that have somewhat limited memory capacity for the moment. Cassandra is eating up all the memory on these nodes. I'm not sure where to look first in terms of reducing the foot print. Keys cached? Compaction? Any hints would be greatly appreciated. Regards, Casey
Re: Reducing memory footprint
On Fri, Mar 4, 2011 at 11:03 AM, Chris Burroughs chris.burrou...@gmail.comwrote: What do you mean by eating up the memory? Resident set size, low memory available to page cache, excessive gc of the jvm's heap? jvm's heap is set for half of the physical memory (1982 MB out of 4G), and jsvc is using 2.9G (73%) of resident memory. Are you saying: that you want a smaller heap and what settings to change to accommodate that, or that you have already set a small heap of x and Cassandra is using significantly more than that? Based on my observation above, the latter. Casey
Re: NegativeArraySizeException with 0.7.2
On Wed, Feb 16, 2011 at 10:01 PM, Nate McCall n...@datastax.com wrote: See the following mail thread: http://www.mail-archive.com/user@cassandra.apache.org/msg10183.html In short, running nodetool compact should clear it up. Thanks for the pointer! I ran nodetool compact on my nodes, and so far it's looking good... Casey
Re: Rows in decreasing order
On Sun, Nov 14, 2010 at 12:50 AM, Jonathan Ellis jbel...@gmail.com wrote: As you may have guessed from the lack of a reversed option on the range slice api, backward scans are not supported. The standard thing to do is load the keys you are interested in as columns to a row. That makes sense. Just needed a different way of thinking about it. Thanks for the clarification. Casey
Rows in decreasing order
Hi, I'm working with Cassandra 0.7.0-beta3. Given rows with keys 1, 2, 3, I'd like to be able to search on key 4 (non-existent) and get row 3 (and possibly the n rows before in a range). I've tried several options, but I keep getting an empty range. What am I missing? Thanks, Casey
