Without knowing what the IOErrors are I would do the following:
nodetool scrub to fix any on disk errors, this will also take a snapshot you can use for rollback. nodetool repair to ensure data is consistent. Hope that helps. ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 16/01/2012, at 7:53 AM, Alexis Lauthier wrote: > I see only one line "JOINING: sleeping 30000 ms for pending range setup". > > Before that, I have a lot of migration INFO messages, only for the > uncompressed families. > > I have currently killed the new node, so a describe cluster shows only one > schema version on the first two nodes. > > [default@unknown] describe cluster; > Cluster Information: > Snitch: org.apache.cassandra.locator.SimpleSnitch > Partitioner: org.apache.cassandra.dht.ByteOrderedPartitioner > Schema versions: > b42595d0-2247-11e1-0000-a0e9ff9ab7bf: [---.36, ---.35] > > UNREACHABLE: [---.56] > > > Also, on one of the old nodes, I have a lot of I/O errors on the data files > for some (but not all) of the compressed families. It began a few days ago. > All "nodetool repair" calls have been blocking since then. > > Any ideas on how I can get the data on the new node, before the old one dies? > > > Thanks, > Alexis Lauthier > > > De : aaron morton <aaron@> > À : user@cassandra.apache.org > Envoyé le : Dimanche 15 Janvier 2012 19h17 > Objet : Re: Compressed families not created on new node > > Sounds like the schema has not fully migrated to the new node. It is applied > to the joining node one change at a time. A quick scan of the changes file > does not find anything fixed after 1.0.3 > > You can check schema versions in the CLI using the describe cluster command. > > Check for errors in the logs with Migration in the text. > > Are you seeing this line a lot in the log ? >> INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) JOINING: >> sleeping 30000 ms for pending range setup > > cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 14/01/2012, at 4:20 AM, Alexis Lauthier wrote: > >> I'm using Cassandra 1.0.3 on a 2 nodes cluster. My schema (with >> replication_factor=2) contains both compressed (with >> sstable_compression=DeflateCompressor) and uncompressed column families. >> >> When bootstrapping a third node, the uncompressed families are created on >> the new node as expected, but the compressed families are not. Only the >> uncompressed families appear in a "show schema", and the new node data size >> is small, which is consistent with the big compressed data not being there. >> >> I'm seeing frequent exceptions in the log : >> >> INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) JOINING: >> sleeping 30000 ms for pending range setup >> ERROR [MutationStage:1] 2012-01-13 14:55:01,511 RowMutationVerbHandler.java >> (line 65) Error in row mutation >> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find >> cfId=1008 >> at >> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129) >> at >> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:401) >> at >> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:409) >> at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:357) >> at >> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46) >> at >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:722) >> >> >> After a few minutes, The column family names are shown instead of their ids >> ("UrlText" is one of the compressed families) : >> >> ERROR [ReadStage:46] 2012-01-13 14:59:33,924 AbstractCassandraDaemon.java >> (line 133) Fatal exception in thread Thread[ReadStage:46,5,main] >> java.lang.IllegalArgumentException: Unknown ColumnFamily UrlText in keyspace >> ptprod >> at org.apache.cassandra.config.Schema.getComparator(Schema.java:226) >> at >> org.apache.cassandra.db.ColumnFamily.getComparatorFor(ColumnFamily.java:300) >> at org.apache.cassandra.db.ReadCommand.getComparator(ReadCommand.java:92) >> at >> org.apache.cassandra.db.SliceByNamesReadCommand.<init>(SliceByNamesReadCommand.java:44) >> at >> org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:106) >> at >> org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:74) >> at >> org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:132) >> at >> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51) >> at >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:722) >> >> >> How can I get the compressed families on the new node ? >> >> Thanks, >> Alexis Lauthier > > >
