[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2240: Attachment: 2240-v5.patch Stepping in while Jonathan takes some well deserved rest and attaching v5 (based on last version attached). This makes the following changes: * In doScrub(), move the first indexFile.readLong() out of the assert. * Fix computation of dataStartFromIndex (was missing th 4 or 8 bytes for the data size). * IndexHelper.defreezeBloomFilter don't leave the file pointer after the bloomFilter for new BF since it reads directly from the file (instead of reading the bytes at once and deserializing from that). Correct this. * Log if a row has been correctly read the first time but index start and size are different from data start and size (since index should then be manually rebuilt). * Do a retry if dataStart == dataStartFromIndex but dataSize != dataSizeFromIndex (in case the row size only has been corrupted). Lastly, a minor remarks: the patch removes a flush in BF.serialize(). Maybe this belongs to another ticket ? nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000873#comment-13000873 ] Sébastien Giroux commented on CASSANDRA-2240: - Thank you Sylvain Unfortunatly, i'm unable to apply v5 patch for some reason, is it a svn or git patch ? nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000880#comment-13000880 ] Sylvain Lebresne commented on CASSANDRA-2240: - It's a git patch, git apply or 'patch -p1 -i 2240-v5.patch' should do the trick (it does here). It's based on current cassandra-0.7 branch. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000895#comment-13000895 ] Sébastien Giroux commented on CASSANDRA-2240: - Yep, that worked. So it's currently running with your patch, and so far I get no output from scrub after running for like 5 mins, I usually get the Scrubbing SSTABLE in the first min so I'm not sure if that's normal. There is the tmp files in the data directory but looks like they are stuck at 0 byte. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000894#comment-13000894 ] Jonathan Ellis commented on CASSANDRA-2240: --- I also backed out the read-directly-from-file change in IndexHelper. Will create a new ticket for that. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2240: -- Attachment: 2240-v6.txt v6 applies Sylvain's fixes to v4 (which mysteriously disappeared from jira) nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000895#comment-13000895 ] Sébastien Giroux edited comment on CASSANDRA-2240 at 3/1/11 2:48 PM: - Yep, that worked. So it's currently running with your patch, and so far I get no output from scrub after running for like 5 mins, I usually get the Scrubbing SSTABLE in the first min so I'm not sure if that's normal. There is the tmp files in the data directory but looks like they are stuck at 0 byte. Oh, looks like I will try v6 now then! Jonathan, v4 is there, you named it 2240.txt :) was (Author: wajam): Yep, that worked. So it's currently running with your patch, and so far I get no output from scrub after running for like 5 mins, I usually get the Scrubbing SSTABLE in the first min so I'm not sure if that's normal. There is the tmp files in the data directory but looks like they are stuck at 0 byte. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000898#comment-13000898 ] Jonathan Ellis commented on CASSANDRA-2240: --- bq. v4 is there, you named it 2240.txt Oops -- that was the original patch, not actually v4. Must have been tired. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000916#comment-13000916 ] Sylvain Lebresne commented on CASSANDRA-2240: - Alright, v6 looks good, +1. Though, we may want to wait to see if it works alright for Sebastien too. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000920#comment-13000920 ] Sébastien Giroux edited comment on CASSANDRA-2240 at 3/1/11 3:36 PM: - I'm testing it as we speak! Log are flooded with this: INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it I'm not sure why it's scrubbing WebsiteWajams CF as I asked for Wajam (nodetool -h 127.0.0.1 scrub Wajam Wajam) ... Either way, this sstable is messed up too so that's fine. Hopefully it won't just do like v4 that ran all night and was still echoing invalid bloom filter for the same sstable in the log this morning, Looks like it was stuck in a infinite loop or something. It looks like it's scrubbing another sstable now so that's already an improvment. Will keep you guys updated! was (Author: wajam): I'm testing it as we speak! Log are flooded with this: INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it I'm not sure why it's scrubbing WebsiteWajams CF as I asked for Wajam (nodetool -h 127.0.0.1 scrub Wajam Wajam) ... Either way, this sstable is messed up too so that's fine. Hopefully it won't just do like v4 that ran all night and was still echoing invalid bloom filter for the same sstable in the log this morning, Looks like it was stuck in a infinite loop or something. It looks like it's scrubbing another sstable now so that's already an improvment. Will keep you guy updated! nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000920#comment-13000920 ] Sébastien Giroux commented on CASSANDRA-2240: - I'm testing it as we speak! Log are flooded with this: INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it INFO 15:31:15,783 Invalid bloom filter in SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-164-Data.db'); will rebuild it I'm not sure why it's scrubbing WebsiteWajams CF as I asked for Wajam (nodetool -h 127.0.0.1 scrub Wajam Wajam) ... Either way, this sstable is messed up too so that's fine. Hopefully it won't just do like v4 that ran all night and was still echoing invalid bloom filter for the same sstable in the log this morning, Looks like it was stuck in a infinite loop or something. It looks like it's scrubbing another sstable now so that's already an improvment. Will keep you guy updated! nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000923#comment-13000923 ] Sébastien Giroux commented on CASSANDRA-2240: - INFO 15:42:52,690 Scrub of SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-155-Data.db') complete: 3271113 in new sstable 3271113 in new sstable... I'm not sure it's 3271113 what... might want to specify! :) I think it's the first time I see this line tho so it's good news. Is there any way the invalid bloom filter, will rebuild it line can only appear once per sstable ? I'm guessing its generating useless IO writing about 1000 of those lines/second in the log! Good work guys, very appreciated :) nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000923#comment-13000923 ] Sébastien Giroux edited comment on CASSANDRA-2240 at 3/1/11 3:50 PM: - INFO 15:42:52,690 Scrub of SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-155-Data.db') complete: 3271113 in new sstable 3271113 in new sstable... I'm not sure it's 3271113 what... Looks like good rows according to the patch, might want to specify! :) I think it's the first time I see this line tho so it's good news. Is there any way the invalid bloom filter, will rebuild it line can only appear once per sstable ? I'm guessing its generating useless IO writing about 1000 of those lines/second in the log! This should be definitly included in 0.7.3 which hopefully will be released soon! Good work guys, very appreciated :) was (Author: wajam): INFO 15:42:52,690 Scrub of SSTableReader(path='/var/lib/cassandra/data/Wajam/WebsiteWajams-f-155-Data.db') complete: 3271113 in new sstable 3271113 in new sstable... I'm not sure it's 3271113 what... might want to specify! :) I think it's the first time I see this line tho so it's good news. Is there any way the invalid bloom filter, will rebuild it line can only appear once per sstable ? I'm guessing its generating useless IO writing about 1000 of those lines/second in the log! Good work guys, very appreciated :) nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1075852 - /cassandra/branches/cassandra-0.7.3/
Author: jbellis Date: Tue Mar 1 15:52:43 2011 New Revision: 1075852 URL: http://svn.apache.org/viewvc?rev=1075852view=rev Log: branch 0.7.3 from the first artifact spin point Added: cassandra/branches/cassandra-0.7.3/ (props changed) - copied from r1074693, cassandra/branches/cassandra-0.7/ Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:ignore (added) +++ svn:ignore Tue Mar 1 15:52:43 2011 @@ -0,0 +1,8 @@ +.classpath +.project +.settings +temp-testng-customsuite.xml +build +build.properties +.idea +out Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:mergeinfo (added) +++ svn:mergeinfo Tue Mar 1 15:52:43 2011 @@ -0,0 +1,9 @@ +/cassandra/branches/cassandra-0.6:922689-1071777 +/cassandra/branches/cassandra-0.7:1026516,1035666,1050269 +/cassandra/branches/cassandra-0.7.0:1053690-1055654 +/cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 +/cassandra/trunk:1026516-1026734,1028929 +/incubator/cassandra/branches/cassandra-0.3:774578-796573 +/incubator/cassandra/branches/cassandra-0.4:810145-834239,834349-834350 +/incubator/cassandra/branches/cassandra-0.5:72-915439 +/incubator/cassandra/branches/cassandra-0.6:911237-922688
svn commit: r1075856 - in /cassandra/branches/cassandra-0.7.3: ./ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/service/
Author: jbellis Date: Tue Mar 1 15:54:50 2011 New Revision: 1075856 URL: http://svn.apache.org/viewvc?rev=1075856view=rev Log: merge r1075198 (fix assert in DatacenterSyncWriteResponseHandler) Modified: cassandra/branches/cassandra-0.7.3/ (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/service/DatacenterSyncWriteResponseHandler.java Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:54:50 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6:922689-1071777 -/cassandra/branches/cassandra-0.7:1026516,1035666,1050269 +/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198 /cassandra/branches/cassandra-0.7.0:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/trunk:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:54:50 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:54:50 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:54:50 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue
svn commit: r1075859 - in /cassandra/branches/cassandra-0.7.3: ./ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/avro/ src/java/org/apache/cassandra/db/filter/ sr
Author: jbellis Date: Tue Mar 1 15:55:49 2011 New Revision: 1075859 URL: http://svn.apache.org/viewvc?rev=1075859view=rev Log: merge r1075231 (cache IColumn and reverse comparators in AbstractType) Modified: cassandra/branches/cassandra-0.7.3/ (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/avro/AvroValidation.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/filter/NamesQueryFilter.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/filter/QueryFilter.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/filter/SliceQueryFilter.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/marshal/AbstractType.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/thrift/ThriftValidation.java Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:55:49 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6:922689-1071777 -/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198 +/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075231 /cassandra/branches/cassandra-0.7.0:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/trunk:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:55:49 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075231 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:55:49 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075231 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:55:49 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198,1075231
svn commit: r1075860 - in /cassandra/branches/cassandra-0.7.3: ./ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/io/sstable/ src/java/org/apache/cassandra/service
Author: jbellis Date: Tue Mar 1 15:56:32 2011 New Revision: 1075860 URL: http://svn.apache.org/viewvc?rev=1075860view=rev Log: merge r1075594 (move file deletions off of scheduledtasks executor) Modified: cassandra/branches/cassandra-0.7.3/ (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/io/sstable/SSTableDeletingReference.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/service/StorageService.java Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:56:32 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6:922689-1071777 -/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075231 +/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075231,1075594 /cassandra/branches/cassandra-0.7.0:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/trunk:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:56:32 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075231 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075231,1075594 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:56:32 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075231 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075231,1075594 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:56:32 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198,1075231 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198,1075231,1075594 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516-1026734,1028929 Propchange:
svn commit: r1075861 - in /cassandra/branches/cassandra-0.7.3: ./ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/hadoop/
Author: jbellis Date: Tue Mar 1 15:58:34 2011 New Revision: 1075861 URL: http://svn.apache.org/viewvc?rev=1075861view=rev Log: merge r1075627 (fix Hadoop ColumnFamilyOutputFormat dropping of mutations) Modified: cassandra/branches/cassandra-0.7.3/ (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordWriter.java Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:58:34 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6:922689-1071777 -/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075231,1075594 +/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075231,1075594,1075627 /cassandra/branches/cassandra-0.7.0:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/trunk:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:58:34 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075231,1075594 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075231,1075594,1075627 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:58:34 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075231,1075594 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075231,1075594,1075627 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 15:58:34 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198,1075231,1075594 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198,1075231,1075594,1075627 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516-1026734,1028929 Propchange:
svn commit: r1075867 - in /cassandra/branches/cassandra-0.7.3: ./ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/avro/ src/java/org/apache/cassandra/db/filter/ sr
Author: jbellis Date: Tue Mar 1 16:01:46 2011 New Revision: 1075867 URL: http://svn.apache.org/viewvc?rev=1075867view=rev Log: revert 1075859/1075231/cache IColumn and reverse comparators in AbstractType Modified: cassandra/branches/cassandra-0.7.3/ (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/avro/AvroValidation.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/filter/NamesQueryFilter.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/filter/QueryFilter.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/filter/SliceQueryFilter.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/marshal/AbstractType.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/thrift/ThriftValidation.java Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 16:01:46 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6:922689-1071777 -/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075231,1075594,1075627 +/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075594,1075627 /cassandra/branches/cassandra-0.7.0:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/trunk:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 16:01:46 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075231,1075594,1075627 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075594,1075627 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 16:01:46 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075231,1075594,1075627 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075594,1075627 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 16:01:46 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516,1035666,1050269,1075198,1075231,1075594,1075627
svn commit: r1075870 - in /cassandra/branches/cassandra-0.7: ./ src/java/org/apache/cassandra/db/ src/java/org/apache/cassandra/io/sstable/ test/unit/org/apache/cassandra/io/
Author: jbellis
Date: Tue Mar 1 16:09:28 2011
New Revision: 1075870
URL: http://svn.apache.org/viewvc?rev=1075870view=rev
Log:
make nodetool scrub more robust
patch by jbellis and slebresne; tested by Sébastien Giroux for CASSANDRA-2240
Modified:
cassandra/branches/cassandra-0.7/CHANGES.txt
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/sstable/IndexHelper.java
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/LazilyCompactedRowTest.java
Modified: cassandra/branches/cassandra-0.7/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1075870r1=1075869r2=1075870view=diff
==
--- cassandra/branches/cassandra-0.7/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.7/CHANGES.txt Tue Mar 1 16:09:28 2011
@@ -22,7 +22,7 @@
* update memtable_throughput to be a long (CASSANDRA-2158)
* fix for compaction and cleanup writing old-format data into new-version
sstable (CASSANDRA-2211, -2216)
- * add nodetool scrub (CASSANDRA-2217)
+ * add nodetool scrub (CASSANDRA-2217, -2240)
* fix sstable2json large-row pagination (CASSANDRA-2188)
* fix EOFing on requests for the last bytes in a file (CASSANDRA-2213)
* fix BufferedRandomAccessFile bugs (CASSANDRA-2218, -2241)
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java?rev=1075870r1=1075869r2=1075870view=diff
==
---
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
(original)
+++
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
Tue Mar 1 16:09:28 2011
@@ -20,6 +20,7 @@ package org.apache.cassandra.db;
import java.io.DataOutput;
import java.io.File;
+import java.io.IOError;
import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.nio.ByteBuffer;
@@ -525,70 +526,132 @@ public class CompactionManager implement
String indexFilename =
sstable.descriptor.filenameFor(Component.PRIMARY_INDEX);
BufferedRandomAccessFile indexFile =
BufferedRandomAccessFile.getUncachingReader(indexFilename);
ByteBuffer nextIndexKey =
ByteBufferUtil.readWithShortLength(indexFile);
-assert indexFile.readLong() == 0;
+{
+// throw away variable so we don't have a side effect in the
assert
+long firstRowPositionFromIndex = indexFile.readLong();
+assert firstRowPositionFromIndex == 0 :
firstRowPositionFromIndex;
+}
SSTableWriter writer = maybeCreateWriter(cfs,
compactionFileLocation, expectedBloomFilterSize, null);
executor.beginCompaction(cfs.columnFamily, new ScrubInfo(dataFile,
sstable));
+int goodRows = 0, badRows = 0;
while (!dataFile.isEOF())
{
long rowStart = dataFile.getFilePointer();
if (logger.isDebugEnabled())
logger.debug(Reading row at + rowStart);
-DecoratedKey key =
SSTableReader.decodeKey(sstable.partitioner, sstable.descriptor,
ByteBufferUtil.readWithShortLength(dataFile));
+
+DecoratedKey key = null;
+long dataSize = -1;
+try
+{
+key = SSTableReader.decodeKey(sstable.partitioner,
sstable.descriptor, ByteBufferUtil.readWithShortLength(dataFile));
+dataSize = sstable.descriptor.hasIntRowSize ?
dataFile.readInt() : dataFile.readLong();
+if (logger.isDebugEnabled())
+logger.debug(String.format(row %s is %s bytes,
ByteBufferUtil.bytesToHex(key.key), dataSize));
+}
+catch (Throwable th)
+{
+throwIfFatal(th);
+// check for null key below
+}
+
ByteBuffer currentIndexKey = nextIndexKey;
-nextIndexKey = indexFile.isEOF() ? null :
ByteBufferUtil.readWithShortLength(indexFile);
-long nextRowPositionFromIndex = indexFile.isEOF() ?
dataFile.length() : indexFile.readLong();
+long nextRowPositionFromIndex;
+try
+{
+nextIndexKey = indexFile.isEOF() ? null :
ByteBufferUtil.readWithShortLength(indexFile);
+nextRowPositionFromIndex = indexFile.isEOF() ?
dataFile.length() : indexFile.readLong();
+
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000931#comment-13000931 ] Sébastien Giroux commented on CASSANDRA-2240: - I did see a few exceptions every now and then. Not sure if you need to do anything about this. Sorry for all these posts, hopefully it help :): 224574- INFO 16:06:40,744 Retrying from row index; data is 245642090 bytes starting at 42 224657- WARN 16:06:40,744 Retry failed too. Skipping to next row (retry's stacktrace follows) 224745-java.io.IOError: java.io.EOFException 224783- at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:117) 224884: at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:610) 224966- at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) 225050- at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) 225131- at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 225202- at java.util.concurrent.FutureTask.run(FutureTask.java:166) 225263- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) 225347- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) 225431- at java.lang.Thread.run(Thread.java:636) 225473-Caused by: java.io.EOFException 225505- at java.io.DataInputStream.readInt(DataInputStream.java:392) 225567- at org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:47) nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2240. --- Resolution: Fixed Reviewer: slebresne committed w/ update to goodRows message and moving the BF/row header messages to debug level thanks! nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000935#comment-13000935 ] Jonathan Ellis commented on CASSANDRA-2240: --- bq. I did see a few exceptions every now and then Did you see any Error reading index file lines before that? That's the only way I can think that it would come up with such a strange row size. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000938#comment-13000938 ] Sébastien Giroux commented on CASSANDRA-2240: - Yep I did see error reading index file before that. Any chance to include this in 0.7.3 release ? I'm sure it would help a ton of people :) nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1427) Optimize loadbalance/move for moves within the current range
[ https://issues.apache.org/jira/browse/CASSANDRA-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1427: --- Attachment: CASSANDRA-1427-v5.patch Optimize loadbalance/move for moves within the current range Key: CASSANDRA-1427 URL: https://issues.apache.org/jira/browse/CASSANDRA-1427 Project: Cassandra Issue Type: Sub-task Components: Core Affects Versions: 0.7 beta 1 Reporter: Nick Bailey Assignee: Pavel Yaskevich Fix For: 0.8 Attachments: CASSANDRA-1427-v2.patch, CASSANDRA-1427-v3.patch, CASSANDRA-1427-v4.patch, CASSANDRA-1427-v5.patch, CASSANDRA-1427.patch Original Estimate: 42h Time Spent: 42h Remaining Estimate: 0h Currently our move/loadbalance operations only implement case 2 of the Ruhl algorithm described at https://issues.apache.org/jira/browse/CASSANDRA-192#action_12713079. We should add functionality to optimize moves that take/give ranges to a node's direct neighbors. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1075888 - in /cassandra/branches/cassandra-0.7: CHANGES.txt src/java/org/apache/cassandra/db/CompactionManager.java
Author: jbellis
Date: Tue Mar 1 16:31:58 2011
New Revision: 1075888
URL: http://svn.apache.org/viewvc?rev=1075888view=rev
Log:
update CHANGES
Modified:
cassandra/branches/cassandra-0.7/CHANGES.txt
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
Modified: cassandra/branches/cassandra-0.7/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1075888r1=1075887r2=1075888view=diff
==
--- cassandra/branches/cassandra-0.7/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.7/CHANGES.txt Tue Mar 1 16:31:58 2011
@@ -1,8 +1,5 @@
0.7.4
* add nodetool join command (CASSANDRA-2160)
- * fix Hadoop ColumnFamilyOutputFormat dropping of mutations
- when batch fills up (CASSANDRA-2255)
- * move file deletions off of scheduledtasks executor (CASSANDRA-2253)
0.7.3
@@ -41,6 +38,9 @@
* fix starting up on Windows when CASSANDRA_HOME contains whitespace
(CASSANDRA-2237)
* add [get|set][row|key]cacheSavePeriod to JMX (CASSANDRA-2100)
+ * fix Hadoop ColumnFamilyOutputFormat dropping of mutations
+ when batch fills up (CASSANDRA-2255)
+ * move file deletions off of scheduledtasks executor (CASSANDRA-2253)
0.7.2
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java?rev=1075888r1=1075887r2=1075888view=diff
==
---
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
(original)
+++
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/CompactionManager.java
Tue Mar 1 16:31:58 2011
@@ -636,7 +636,7 @@ public class CompactionManager implement
cfs.replaceCompactedSSTables(Arrays.asList(sstable),
Arrays.asList(newSstable));
logger.info(Scrub of + sstable + complete: + goodRows +
rows in new sstable);
if (badRows 0)
-logger.warn(Unable to recover + badRows + that were
skipped. You can attempt manual recovery from the pre-scrub snapshot. You can
also run nodetool repair to transfer the data from a healthy replica, if any);
+logger.warn(Unable to recover + badRows + rows that
were skipped. You can attempt manual recovery from the pre-scrub snapshot.
You can also run nodetool repair to transfer the data from a healthy replica,
if any);
}
else
{
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000944#comment-13000944 ] Jonathan Ellis commented on CASSANDRA-2240: --- Yes, we'll get this into 0.7.3. nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.4 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1075890 - in /cassandra/branches/cassandra-0.7.3: ./ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/db/ src/java/org/apache/cassandra/io/sstable/ tes
Author: jbellis Date: Tue Mar 1 16:33:04 2011 New Revision: 1075890 URL: http://svn.apache.org/viewvc?rev=1075890view=rev Log: merge 1075870 (make nodetool scrub more robust) Modified: cassandra/branches/cassandra-0.7.3/ (props changed) cassandra/branches/cassandra-0.7.3/CHANGES.txt cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/db/CompactionManager.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/io/sstable/IndexHelper.java cassandra/branches/cassandra-0.7.3/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java cassandra/branches/cassandra-0.7.3/test/unit/org/apache/cassandra/io/LazilyCompactedRowTest.java Propchange: cassandra/branches/cassandra-0.7.3/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 16:33:04 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6:922689-1071777 -/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075594,1075627 +/cassandra/branches/cassandra-0.7:1026516,1035666,1050269,1075198,1075594,1075627,1075870 /cassandra/branches/cassandra-0.7.0:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/trunk:1026516-1026734,1028929 Modified: cassandra/branches/cassandra-0.7.3/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7.3/CHANGES.txt?rev=1075890r1=1075889r2=1075890view=diff == --- cassandra/branches/cassandra-0.7.3/CHANGES.txt (original) +++ cassandra/branches/cassandra-0.7.3/CHANGES.txt Tue Mar 1 16:33:04 2011 @@ -15,7 +15,7 @@ * update memtable_throughput to be a long (CASSANDRA-2158) * fix for compaction and cleanup writing old-format data into new-version sstable (CASSANDRA-2211, -2216) - * add nodetool scrub (CASSANDRA-2217) + * add nodetool scrub (CASSANDRA-2217, -2240) * fix sstable2json large-row pagination (CASSANDRA-2188) * fix EOFing on requests for the last bytes in a file (CASSANDRA-2213) * fix BufferedRandomAccessFile bugs (CASSANDRA-2218, -2241) @@ -34,6 +34,9 @@ * fix starting up on Windows when CASSANDRA_HOME contains whitespace (CASSANDRA-2237) * add [get|set][row|key]cacheSavePeriod to JMX (CASSANDRA-2100) + * fix Hadoop ColumnFamilyOutputFormat dropping of mutations + when batch fills up (CASSANDRA-2255) + * move file deletions off of scheduledtasks executor (CASSANDRA-2253) 0.7.2 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 16:33:04 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075594,1075627 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516,1035666,1050269,1075198,1075594,1075627,1075870 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1026734,1028929 Propchange: cassandra/branches/cassandra-0.7.3/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Mar 1 16:33:04 2011 @@ -1,5 +1,5 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1071777 -/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075594,1075627 +/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516,1035666,1050269,1075198,1075594,1075627,1075870
[jira] Updated: (CASSANDRA-2255) ColumnFamilyOutputFormat drops mutations when batches fill up.
[ https://issues.apache.org/jira/browse/CASSANDRA-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2255: -- Fix Version/s: (was: 0.7.4) 0.7.3 ColumnFamilyOutputFormat drops mutations when batches fill up. -- Key: CASSANDRA-2255 URL: https://issues.apache.org/jira/browse/CASSANDRA-2255 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 0.7.2, 0.8 Reporter: Eldon Stegall Assignee: Jeremy Hanna Fix For: 0.7.3 Attachments: 0001_Stop_dropping_mutations.txt, 2255-patch-2.txt queue.poll() takes a mutation, but then the batch is already full, so the while loop exits, ant the mutation we just got is dropped. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2240: -- Fix Version/s: (was: 0.7.4) 0.7.3 nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.3 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2253) Gossiper Starvation
[ https://issues.apache.org/jira/browse/CASSANDRA-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2253: -- Fix Version/s: (was: 0.7.4) 0.7.3 Gossiper Starvation --- Key: CASSANDRA-2253 URL: https://issues.apache.org/jira/browse/CASSANDRA-2253 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.0 Environment: linux, windows Reporter: Mikael Sitruk Assignee: Mikael Sitruk Fix For: 0.7.3 Attachments: CASSANDRA-0.7-2253.txt Original Estimate: 2h Remaining Estimate: 2h Gossiper periodic task will get into starvation in case large sstable files need to be deleted. Indeed the SSTableDeletingReference uses the same scheduledTasks pool (from StorageService) as the Gossiper and other periodic tasks, but the gossiper tasks should run each second to assure correct cluster status (liveness of nodes). In case of large sstable files to be deleted (several GB) the delete operation can take more than 30 sec, thus making the whole cluster going into a wrong state where nodes are marked as not living while they are! This will lead to unneeded additional load like hinted hand off, wrong cluster state, increase in latency. One of the possible solution is to use a separate pool for periodic and non periodic tasks. I've implemented such change and it resolves the problem. I can provide a patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000953#comment-13000953 ] Hudson commented on CASSANDRA-2240: --- Integrated in Cassandra-0.7 #336 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/336/]) make nodetool scrub more robust patch by jbellis and slebresne; tested by Sébastien Giroux for CASSANDRA-2240 nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.3 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1427) Optimize loadbalance/move for moves within the current range
[ https://issues.apache.org/jira/browse/CASSANDRA-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1427: --- Attachment: (was: CASSANDRA-1427-v5.patch) Optimize loadbalance/move for moves within the current range Key: CASSANDRA-1427 URL: https://issues.apache.org/jira/browse/CASSANDRA-1427 Project: Cassandra Issue Type: Sub-task Components: Core Affects Versions: 0.7 beta 1 Reporter: Nick Bailey Assignee: Pavel Yaskevich Fix For: 0.8 Attachments: CASSANDRA-1427-v2.patch, CASSANDRA-1427-v3.patch, CASSANDRA-1427-v4.patch, CASSANDRA-1427.patch Original Estimate: 42h Time Spent: 42h Remaining Estimate: 0h Currently our move/loadbalance operations only implement case 2 of the Ruhl algorithm described at https://issues.apache.org/jira/browse/CASSANDRA-192#action_12713079. We should add functionality to optimize moves that take/give ranges to a node's direct neighbors. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1427) Optimize loadbalance/move for moves within the current range
[ https://issues.apache.org/jira/browse/CASSANDRA-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1427: --- Attachment: CASSANDRA-1427-v5.patch Optimize loadbalance/move for moves within the current range Key: CASSANDRA-1427 URL: https://issues.apache.org/jira/browse/CASSANDRA-1427 Project: Cassandra Issue Type: Sub-task Components: Core Affects Versions: 0.7 beta 1 Reporter: Nick Bailey Assignee: Pavel Yaskevich Fix For: 0.8 Attachments: CASSANDRA-1427-v2.patch, CASSANDRA-1427-v3.patch, CASSANDRA-1427-v4.patch, CASSANDRA-1427-v5.patch, CASSANDRA-1427.patch Original Estimate: 42h Time Spent: 42h Remaining Estimate: 0h Currently our move/loadbalance operations only implement case 2 of the Ruhl algorithm described at https://issues.apache.org/jira/browse/CASSANDRA-192#action_12713079. We should add functionality to optimize moves that take/give ranges to a node's direct neighbors. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1709) CQL keyspace and column family management
[ https://issues.apache.org/jira/browse/CASSANDRA-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Evans updated CASSANDRA-1709: -- Attachment: v1-0003-updated-doco-for-CQL-DROPs.txt v1-0002-system-tests-for-CQL-DROPs.txt v1-0001-CASSANDRA-1709-CQL-DROP-implementations.txt CQL keyspace and column family management - Key: CASSANDRA-1709 URL: https://issues.apache.org/jira/browse/CASSANDRA-1709 Project: Cassandra Issue Type: Sub-task Components: API Affects Versions: 0.8 Reporter: Eric Evans Assignee: Eric Evans Priority: Minor Labels: cql Fix For: 0.8 Attachments: v1-0001-CASSANDRA-1709-CQL-DROP-implementations.txt, v1-0001-CASSANDRA-1709-CREATE-INDEX-implementation.txt, v1-0002-system-tests-for-CQL-DROPs.txt, v1-0002-system-tests-for-CREATE-INDEX.txt, v1-0003-documentation-update-for-CREATE-INDEX.txt, v1-0003-updated-doco-for-CQL-DROPs.txt Original Estimate: 0h Remaining Estimate: 0h CQL specification and implementation for schema management. This corresponds to the following RPC methods: * system_add_column_family() * system_add_keyspace() * system_drop_keyspace() * system_update_keyspace() * system_update_columnfamily() -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1709) CQL keyspace and column family management
[ https://issues.apache.org/jira/browse/CASSANDRA-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Evans updated CASSANDRA-1709: -- Attachment: (was: v1-0001-CASSANDRA-1709-CREATE-INDEX-implementation.txt) CQL keyspace and column family management - Key: CASSANDRA-1709 URL: https://issues.apache.org/jira/browse/CASSANDRA-1709 Project: Cassandra Issue Type: Sub-task Components: API Affects Versions: 0.8 Reporter: Eric Evans Assignee: Eric Evans Priority: Minor Labels: cql Fix For: 0.8 Attachments: v1-0001-CASSANDRA-1709-CQL-DROP-implementations.txt, v1-0002-system-tests-for-CQL-DROPs.txt, v1-0003-updated-doco-for-CQL-DROPs.txt Original Estimate: 0h Remaining Estimate: 0h CQL specification and implementation for schema management. This corresponds to the following RPC methods: * system_add_column_family() * system_add_keyspace() * system_drop_keyspace() * system_update_keyspace() * system_update_columnfamily() -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1709) CQL keyspace and column family management
[ https://issues.apache.org/jira/browse/CASSANDRA-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Evans updated CASSANDRA-1709: -- Attachment: (was: v1-0002-system-tests-for-CREATE-INDEX.txt) CQL keyspace and column family management - Key: CASSANDRA-1709 URL: https://issues.apache.org/jira/browse/CASSANDRA-1709 Project: Cassandra Issue Type: Sub-task Components: API Affects Versions: 0.8 Reporter: Eric Evans Assignee: Eric Evans Priority: Minor Labels: cql Fix For: 0.8 Attachments: v1-0001-CASSANDRA-1709-CQL-DROP-implementations.txt, v1-0002-system-tests-for-CQL-DROPs.txt, v1-0003-updated-doco-for-CQL-DROPs.txt Original Estimate: 0h Remaining Estimate: 0h CQL specification and implementation for schema management. This corresponds to the following RPC methods: * system_add_column_family() * system_add_keyspace() * system_drop_keyspace() * system_update_keyspace() * system_update_columnfamily() -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1709) CQL keyspace and column family management
[
https://issues.apache.org/jira/browse/CASSANDRA-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13000978#comment-13000978
]
Eric Evans commented on CASSANDRA-1709:
---
Patches attached for {{DROP}} (keyspace and column family).
CQL keyspace and column family management
-
Key: CASSANDRA-1709
URL: https://issues.apache.org/jira/browse/CASSANDRA-1709
Project: Cassandra
Issue Type: Sub-task
Components: API
Affects Versions: 0.8
Reporter: Eric Evans
Assignee: Eric Evans
Priority: Minor
Labels: cql
Fix For: 0.8
Attachments: v1-0001-CASSANDRA-1709-CQL-DROP-implementations.txt,
v1-0002-system-tests-for-CQL-DROPs.txt, v1-0003-updated-doco-for-CQL-DROPs.txt
Original Estimate: 0h
Remaining Estimate: 0h
CQL specification and implementation for schema management.
This corresponds to the following RPC methods:
* system_add_column_family()
* system_add_keyspace()
* system_drop_keyspace()
* system_update_keyspace()
* system_update_columnfamily()
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1427) Optimize loadbalance/move for moves within the current range
[ https://issues.apache.org/jira/browse/CASSANDRA-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1427: --- Attachment: (was: CASSANDRA-1427-v5.patch) Optimize loadbalance/move for moves within the current range Key: CASSANDRA-1427 URL: https://issues.apache.org/jira/browse/CASSANDRA-1427 Project: Cassandra Issue Type: Sub-task Components: Core Affects Versions: 0.7 beta 1 Reporter: Nick Bailey Assignee: Pavel Yaskevich Fix For: 0.8 Attachments: CASSANDRA-1427-v2.patch, CASSANDRA-1427-v3.patch, CASSANDRA-1427-v4.patch, CASSANDRA-1427-v5.patch, CASSANDRA-1427.patch Original Estimate: 42h Time Spent: 42h Remaining Estimate: 0h Currently our move/loadbalance operations only implement case 2 of the Ruhl algorithm described at https://issues.apache.org/jira/browse/CASSANDRA-192#action_12713079. We should add functionality to optimize moves that take/give ranges to a node's direct neighbors. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1427) Optimize loadbalance/move for moves within the current range
[ https://issues.apache.org/jira/browse/CASSANDRA-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-1427: --- Attachment: CASSANDRA-1427-v5.patch Optimize loadbalance/move for moves within the current range Key: CASSANDRA-1427 URL: https://issues.apache.org/jira/browse/CASSANDRA-1427 Project: Cassandra Issue Type: Sub-task Components: Core Affects Versions: 0.7 beta 1 Reporter: Nick Bailey Assignee: Pavel Yaskevich Fix For: 0.8 Attachments: CASSANDRA-1427-v2.patch, CASSANDRA-1427-v3.patch, CASSANDRA-1427-v4.patch, CASSANDRA-1427-v5.patch, CASSANDRA-1427.patch Original Estimate: 42h Time Spent: 42h Remaining Estimate: 0h Currently our move/loadbalance operations only implement case 2 of the Ruhl algorithm described at https://issues.apache.org/jira/browse/CASSANDRA-192#action_12713079. We should add functionality to optimize moves that take/give ranges to a node's direct neighbors. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1075976 - /cassandra/branches/cassandra-0.7.3/debian/changelog
Author: eevans Date: Tue Mar 1 18:54:47 2011 New Revision: 1075976 URL: http://svn.apache.org/viewvc?rev=1075976view=rev Log: update changelog timestamp Modified: cassandra/branches/cassandra-0.7.3/debian/changelog Modified: cassandra/branches/cassandra-0.7.3/debian/changelog URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7.3/debian/changelog?rev=1075976r1=1075975r2=1075976view=diff == --- cassandra/branches/cassandra-0.7.3/debian/changelog (original) +++ cassandra/branches/cassandra-0.7.3/debian/changelog Tue Mar 1 18:54:47 2011 @@ -2,7 +2,7 @@ cassandra (0.7.3) unstable; urgency=low * New stable point release. - -- Eric Evans eev...@apache.org Fri, 25 Feb 2011 14:20:50 -0600 + -- Eric Evans eev...@apache.org Tue, 01 Mar 2011 12:55:13 -0600 cassandra (0.7.1) unstable; urgency=low
svn commit: r1075977 - /cassandra/branches/cassandra-0.7.3/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java
Author: eevans Date: Tue Mar 1 18:55:05 2011 New Revision: 1075977 URL: http://svn.apache.org/viewvc?rev=1075977view=rev Log: prepend missing license header Modified: cassandra/branches/cassandra-0.7.3/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java Modified: cassandra/branches/cassandra-0.7.3/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7.3/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java?rev=1075977r1=1075976r2=1075977view=diff == --- cassandra/branches/cassandra-0.7.3/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java (original) +++ cassandra/branches/cassandra-0.7.3/test/unit/org/apache/cassandra/io/sstable/DescriptorTest.java Tue Mar 1 18:55:05 2011 @@ -1,4 +1,25 @@ package org.apache.cassandra.io.sstable; +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * + */ + import java.io.File;
[jira] Commented: (CASSANDRA-2240) nodetool scrub hangs or throws an exception
[ https://issues.apache.org/jira/browse/CASSANDRA-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001088#comment-13001088 ] Yaniv Kunda commented on CASSANDRA-2240: Thank you so much, my sstables are finally clean! nodetool scrub hangs or throws an exception --- Key: CASSANDRA-2240 URL: https://issues.apache.org/jira/browse/CASSANDRA-2240 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 0.7.3 Environment: using build #314 from hudson Reporter: Yaniv Kunda Assignee: Jonathan Ellis Fix For: 0.7.3 Attachments: 2240-v2.txt, 2240-v3.txt, 2240-v5.patch, 2240-v6.txt, 2240.txt, 2240.txt, exception2.txt, jstack1.txt, signatureBuckets-f-104.tar.gz, system.log.2.gz, system.log.gz, test-0.6.x-tables.tar.gz, userChannelFilter-f-210.tar.gz trying to run nodetool scrub hung or (only happened one time) threw the following exception: ERROR [CompactionExecutor:1] 2011-02-28 10:26:26,620 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.AssertionError at org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:62) at org.apache.cassandra.io.sstable.SSTableReader.decodeKey(SSTableReader.java:627) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:538) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:55) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:194) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1076026 - /cassandra/branches/cassandra-0.7/build.xml
Author: jbellis
Date: Tue Mar 1 21:35:03 2011
New Revision: 1076026
URL: http://svn.apache.org/viewvc?rev=1076026view=rev
Log:
add cobertura instrumented classes to clean target
Modified:
cassandra/branches/cassandra-0.7/build.xml
Modified: cassandra/branches/cassandra-0.7/build.xml
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/build.xml?rev=1076026r1=1076025r2=1076026view=diff
==
--- cassandra/branches/cassandra-0.7/build.xml (original)
+++ cassandra/branches/cassandra-0.7/build.xml Tue Mar 1 21:35:03 2011
@@ -105,6 +105,7 @@
target name=clean description=Remove all locally created artifacts
delete dir=${build.test.dir} /
delete dir=${build.classes} /
+delete dir=${cobertura.classes.dir} /
delete dir=${build.src.gen-java} /
delete dir=${interface.avro.dir} /
delete file=${build.dir}/internode.avpr /
[jira] Created: (CASSANDRA-2258) service.SerializationsTest failes under cobertura
service.SerializationsTest failes under cobertura
-
Key: CASSANDRA-2258
URL: https://issues.apache.org/jira/browse/CASSANDRA-2258
Project: Cassandra
Issue Type: Bug
Components: Tests
Reporter: Jonathan Ellis
Assignee: Gary Dusbabek
Priority: Minor
Fix For: 0.7.4
ant codecoverage -Dtest.name=SerializationsTest gives
{noformat}
[junit] Testcase:
testTreeResponseRead(org.apache.cassandra.service.SerializationsTest):
Caused an ERROR
[junit] java.io.InvalidClassException:
org.apache.cassandra.dht.BigIntegerToken; local class incompatible: stream
classdesc serialVersionUID = -5833589141319293006, local class serialVersionUID
= 2280189098581028124
[junit] java.lang.RuntimeException: java.io.InvalidClassException:
org.apache.cassandra.dht.BigIntegerToken; local class incompatible: stream
classdesc serialVersionUID = -5833589141319293006, local class serialVersionUID
= 2280189098581028124
[junit] at
org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.deserialize(AntiEntropyService.java:634)
[junit] at
org.apache.cassandra.service.SerializationsTest.testTreeResponseRead(SerializationsTest.java:90)
[junit] Caused by: java.io.InvalidClassException:
org.apache.cassandra.dht.BigIntegerToken; local class incompatible: stream
classdesc serialVersionUID = -5833589141319293006, local class serialVersionUID
= 2280189098581028124
[junit] at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:562)
[junit] at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
[junit] at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
[junit] at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
[junit] at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
[junit] at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
[junit] at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
[junit] at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
[junit] at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
[junit] at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
[junit] at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
[junit] at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
[junit] at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
[junit] at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
[junit] at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
[junit] at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
[junit] at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
[junit] at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
[junit] at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
[junit] at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
[junit] at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
[junit] at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
[junit] at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
[junit] at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
[junit] at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
[junit] at
java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
[junit] at
org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.deserialize(AntiEntropyService.java:630)
{noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (CASSANDRA-2259) column values are only being validated in insert()
column values are only being validated in insert() -- Key: CASSANDRA-2259 URL: https://issues.apache.org/jira/browse/CASSANDRA-2259 Project: Cassandra Issue Type: Bug Components: Core Reporter: Eric Evans Fix For: 0.8 insert() is the only code path that currently results in validate() being called for column values; it is possible to write invalid column values using batch_mutate() -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1076041 - /cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
Author: jbellis
Date: Tue Mar 1 22:44:06 2011
New Revision: 1076041
URL: http://svn.apache.org/viewvc?rev=1076041view=rev
Log:
improve BufferedRAFTest
Modified:
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
Modified:
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java?rev=1076041r1=1076040r2=1076041view=diff
==
---
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
(original)
+++
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
Tue Mar 1 22:44:06 2011
@@ -1,6 +1,4 @@
-package org.apache.cassandra.io.util;
-/*
- *
+/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -8,32 +6,147 @@ package org.apache.cassandra.io.util;
* to you under the Apache License, Version 2.0 (the
* License); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
- *
+ *
* http://www.apache.org/licenses/LICENSE-2.0
- *
+ *
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
- *
+ *
*/
+package org.apache.cassandra.io.util;
+import org.apache.cassandra.utils.ByteBufferUtil;
import java.io.EOFException;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.ClosedChannelException;
import java.util.Arrays;
import java.util.concurrent.Callable;
-import org.junit.Test;
-
+import static org.junit.Assert.assertTrue;
import static org.junit.Assert.assertEquals;
+import org.junit.Test;
+
public class BufferedRandomAccessFileTest
{
+@Test
+public void testReadAndWrite() throws Exception
+{
+BufferedRandomAccessFile file = createTempFile(braf);
+
+// writting string of data to the file
+byte[] data = Hello.getBytes();
+file.write(data);
+assertEquals(file.length(), data.length);
+assertEquals(file.getFilePointer(), data.length);
+
+// reading small amount of data from file, this is handled by initial
buffer
+file.seek(0);
+byte[] buffer = new byte[data.length];
+assertEquals(file.read(buffer), data.length);
+assertTrue(Arrays.equals(buffer, data)); // we read exactly what we
wrote
+assertEquals(file.read(), -1); // nothing more to read EOF
+assert file.bytesRemaining() == 0 file.isEOF();
+
+// writing buffer bigger than page size, which will trigger reBuffer()
+byte[] bigData = new byte[BufferedRandomAccessFile.DEFAULT_BUFFER_SIZE
+ 10];
+
+for (int i = 0; i bigData.length; i++)
+bigData[i] = 'd';
+
+long initialPosition = file.getFilePointer();
+file.write(bigData); // writing data
+assertEquals(file.getFilePointer(), initialPosition + bigData.length);
+assertEquals(file.length(), initialPosition + bigData.length); // file
size should equals to last position
+
+// reading written buffer
+file.seek(initialPosition); // back to initial (before write) position
+data = new byte[bigData.length];
+long sizeRead = 0;
+for (int i = 0; i data.length; i++)
+{
+data[i] = (byte) file.read(); // this will trigger reBuffer()
+sizeRead++;
+}
+
+assertEquals(sizeRead, data.length); // read exactly data.length bytes
+assertEquals(file.getFilePointer(), initialPosition + data.length);
+assertEquals(file.length(), initialPosition + bigData.length);
+assertTrue(Arrays.equals(bigData, data));
+assert file.bytesRemaining() == 0 file.isEOF(); // we are at the of
the file
+
+// test readBytes(int) method
+file.seek(0);
+ByteBuffer fileContent = file.readBytes((int) file.length());
+assertEquals(fileContent.limit(), file.length());
+assert ByteBufferUtil.string(fileContent).equals(Hello + new
String(bigData));
+
+// read the same buffer but using readFully(int)
+data = new byte[bigData.length];
+file.seek(initialPosition);
+file.readFully(data);
+assert file.bytesRemaining() == 0 file.isEOF(); // we should be at
EOF
+
svn commit: r1076042 - /cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
Author: jbellis
Date: Tue Mar 1 22:45:02 2011
New Revision: 1076042
URL: http://svn.apache.org/viewvc?rev=1076042view=rev
Log:
revert last pending cleanup
Modified:
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
Modified:
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java?rev=1076042r1=1076041r2=1076042view=diff
==
---
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
(original)
+++
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/io/util/BufferedRandomAccessFileTest.java
Tue Mar 1 22:45:02 2011
@@ -1,4 +1,6 @@
-/**
+package org.apache.cassandra.io.util;
+/*
+ *
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -6,147 +8,32 @@
* to you under the Apache License, Version 2.0 (the
* License); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
- *
+ *
* http://www.apache.org/licenses/LICENSE-2.0
- *
+ *
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
- *
+ *
*/
-package org.apache.cassandra.io.util;
-import org.apache.cassandra.utils.ByteBufferUtil;
import java.io.EOFException;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.channels.ClosedChannelException;
import java.util.Arrays;
import java.util.concurrent.Callable;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.assertEquals;
-
import org.junit.Test;
+import static org.junit.Assert.assertEquals;
+
public class BufferedRandomAccessFileTest
{
-@Test
-public void testReadAndWrite() throws Exception
-{
-BufferedRandomAccessFile file = createTempFile(braf);
-
-// writting string of data to the file
-byte[] data = Hello.getBytes();
-file.write(data);
-assertEquals(file.length(), data.length);
-assertEquals(file.getFilePointer(), data.length);
-
-// reading small amount of data from file, this is handled by initial
buffer
-file.seek(0);
-byte[] buffer = new byte[data.length];
-assertEquals(file.read(buffer), data.length);
-assertTrue(Arrays.equals(buffer, data)); // we read exactly what we
wrote
-assertEquals(file.read(), -1); // nothing more to read EOF
-assert file.bytesRemaining() == 0 file.isEOF();
-
-// writing buffer bigger than page size, which will trigger reBuffer()
-byte[] bigData = new byte[BufferedRandomAccessFile.DEFAULT_BUFFER_SIZE
+ 10];
-
-for (int i = 0; i bigData.length; i++)
-bigData[i] = 'd';
-
-long initialPosition = file.getFilePointer();
-file.write(bigData); // writing data
-assertEquals(file.getFilePointer(), initialPosition + bigData.length);
-assertEquals(file.length(), initialPosition + bigData.length); // file
size should equals to last position
-
-// reading written buffer
-file.seek(initialPosition); // back to initial (before write) position
-data = new byte[bigData.length];
-long sizeRead = 0;
-for (int i = 0; i data.length; i++)
-{
-data[i] = (byte) file.read(); // this will trigger reBuffer()
-sizeRead++;
-}
-
-assertEquals(sizeRead, data.length); // read exactly data.length bytes
-assertEquals(file.getFilePointer(), initialPosition + data.length);
-assertEquals(file.length(), initialPosition + bigData.length);
-assertTrue(Arrays.equals(bigData, data));
-assert file.bytesRemaining() == 0 file.isEOF(); // we are at the of
the file
-
-// test readBytes(int) method
-file.seek(0);
-ByteBuffer fileContent = file.readBytes((int) file.length());
-assertEquals(fileContent.limit(), file.length());
-assert ByteBufferUtil.string(fileContent).equals(Hello + new
String(bigData));
-
-// read the same buffer but using readFully(int)
-data = new byte[bigData.length];
-file.seek(initialPosition);
-file.readFully(data);
-assert file.bytesRemaining() == 0 file.isEOF(); // we should be at
EOF
-assertTrue(Arrays.equals(bigData, data));
-
[jira] Commented: (CASSANDRA-2242) Improve BRAFTest
[ https://issues.apache.org/jira/browse/CASSANDRA-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001163#comment-13001163 ] Jonathan Ellis commented on CASSANDRA-2242: --- Let's not use @Test(expected=X) in tests that also contain other assertions -- the test will pass if an exception is thrown at the wrong time, too. Improve BRAFTest Key: CASSANDRA-2242 URL: https://issues.apache.org/jira/browse/CASSANDRA-2242 Project: Cassandra Issue Type: Test Components: Core Reporter: Jonathan Ellis Assignee: Pavel Yaskevich Fix For: 0.7.4 Attachments: CASSANDRA-2242.patch BRAF is insufficiently tested (see CASSANDRA-2241). I'd like to get this up to 100% line coverage. (Not sure what it's actually at right now, since ant codecoverage doesn't work.) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2256) BRAF assertion error
[
https://issues.apache.org/jira/browse/CASSANDRA-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-2256:
--
Fix Version/s: (was: 0.7.3)
0.7.4
BRAF assertion error
Key: CASSANDRA-2256
URL: https://issues.apache.org/jira/browse/CASSANDRA-2256
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.3
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
Fix For: 0.7.4
While investigating CASSANDRA-2240 I ran into this:
{noformat}
java.lang.AssertionError
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java\
:230)
at java.io.RandomAccessFile.readByte(RandomAccessFile.java:589)
at
org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:273)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:284)
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:539)
{noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CASSANDRA-2217) nodetool scrub
[ https://issues.apache.org/jira/browse/CASSANDRA-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2217. --- Resolution: Fixed see CASSANDRA-2240 for followup nodetool scrub -- Key: CASSANDRA-2217 URL: https://issues.apache.org/jira/browse/CASSANDRA-2217 Project: Cassandra Issue Type: Bug Components: Tools Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 0.7.3 Attachments: 0001-CASSANDRA-2217-nodetool-scrub-with-broken-sstable-test.txt, 2217-v2.txt, 2217-v3.txt, 2217-v4.txt, 2217-v5.txt, 2217-v7.txt, 2217.txt this is the force-deserialization tool to repair CASSANDRA-2211 and CASSANDRA-2216. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (CASSANDRA-2256) BRAF assertion error
[
https://issues.apache.org/jira/browse/CASSANDRA-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis reassigned CASSANDRA-2256:
-
Assignee: Pavel Yaskevich (was: Jonathan Ellis)
BRAF assertion error
Key: CASSANDRA-2256
URL: https://issues.apache.org/jira/browse/CASSANDRA-2256
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.3
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
Fix For: 0.7.4
While investigating CASSANDRA-2240 I ran into this:
{noformat}
java.lang.AssertionError
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java\
:230)
at java.io.RandomAccessFile.readByte(RandomAccessFile.java:589)
at
org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:273)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:284)
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:539)
{noformat}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2244) repair doesn't handle secondary indexes
[
https://issues.apache.org/jira/browse/CASSANDRA-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001183#comment-13001183
]
Brandon Williams commented on CASSANDRA-2244:
-
I added some logging, it does do something after streaming:
{noformat}
INFO 23:27:32,050 Opening /var/lib/cassandra/data/Keyspace1/Standard1-f-1
INFO 23:27:34,843 Opening /var/lib/cassandra/data/Keyspace1/Standard1-f-2
INFO 23:27:35,232 Opening /var/lib/cassandra/data/Keyspace1/Standard1-f-3
INFO 23:27:35,246 Building index for ColumnFamilyStore(table='Keyspace1',
columnFamily='Standard1')
[SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-f-1-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1-f-3-Data.db')]
[java.nio.HeapByteBuffer[pos=0 lim=2 cap=2]]
INFO 23:29:53,448 Finished streaming session 19283352579402862 from
/10.179.65.102
{noformat}
During the ~2.5 minutes between the last 2 lines it appeared to generate the
index, however flushing afterwards shows no index was generated:
{noformat}
INFO 23:32:06,591 switching in a fresh Memtable for LocationInfo at
CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1299022015229.log',
position=12763)
INFO 23:32:06,592 Enqueuing flush of Memtable-LocationInfo@1459852990(35
bytes, 1 operations)
INFO 23:32:06,593 Writing Memtable-LocationInfo@1459852990(35 bytes, 1
operations)
INFO 23:32:06,623 Completed flushing
/var/lib/cassandra/data/system/LocationInfo-f-3-Data.db (89 bytes)
{noformat}
repair doesn't handle secondary indexes
---
Key: CASSANDRA-2244
URL: https://issues.apache.org/jira/browse/CASSANDRA-2244
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.0
Reporter: Brandon Williams
Assignee: Jonathan Ellis
Fix For: 0.7.4
The repaired node neither receives indexes from the replicas, nor does it
generate them afterwards.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of HadoopSupport by jeremyhanna
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The HadoopSupport page has been changed by jeremyhanna. The comment on this change is: Updating with more information about the virtual datacenter concept and more configuration help.. http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=19rev2=20 -- * [[#Pig|Pig Support]] * [[#Hive|Hive Support]] * [[#ClusterConfig|Cluster Configuration]] + * [[#Support|Support]] Anchor(Overview) == Overview == - Cassandra version 0.6 and later enable certain Hadoop functionality against Cassandra's data store. Specifically, support has been added for [[http://hadoop.apache.org/mapreduce/|MapReduce]], [[http://hadoop.apache.org/pig/|Pig]] and [[http://hive.apache.org/|Hive]]. + Cassandra 0.6+ enables certain Hadoop functionality against Cassandra's data store. Specifically, support has been added for [[http://hadoop.apache.org/mapreduce/|MapReduce]], [[http://pig.apache.org|Pig]] and [[http://hive.apache.org/|Hive]]. [[#Top|Top]] @@ -21, +22 @@ == MapReduce == Input from Cassandra - Cassandra 0.6 (and later) adds support for retrieving data from Cassandra. This is based on implementations of [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/InputSplit.html|InputSplit]], [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/InputFormat.html|InputFormat]], and [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/RecordReader.html|RecordReader]] so that Hadoop !MapReduce jobs can retrieve data from Cassandra. For an example of how this works, see the contrib/word_count example in 0.6 or later. Cassandra rows or row fragments (that is, pairs of key + `SortedMap` of columns) are input to Map tasks for processing by your job, as specified by a `SlicePredicate` that describes which columns to fetch from each row. + Cassandra 0.6+ adds support for retrieving data from Cassandra. This is based on implementations of [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/InputSplit.html|InputSplit]], [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/InputFormat.html|InputFormat]], and [[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/RecordReader.html|RecordReader]] so that Hadoop !MapReduce jobs can retrieve data from Cassandra. For an example of how this works, see the contrib/word_count example in 0.6 or later. Cassandra rows or row fragments (that is, pairs of key + `SortedMap` of columns) are input to Map tasks for processing by your job, as specified by a `SlicePredicate` that describes which columns to fetch from each row. Here's how this looks in the word_count example, which selects just one configurable columnName from each row: @@ -31, +32 @@ ConfigHelper.setSlicePredicate(job.getConfiguration(), predicate); }}} - As of 0.7, configuration for Hadoop no longer resides in your job's specific storage-conf.xml. See the `README` in the word_count and pig contrib modules for more details. + As of 0.7, configuration for Hadoop no longer resides in your job's specific storage-conf.xml. See the `README` in the `word_count` and `pig` contrib modules for more details. Output To Cassandra - As of 0.7, there is be a basic mechanism included in Cassandra for outputting data to Cassandra. The contrib/word_count example in 0.7 contains two reducers - one for outputting data to the filesystem (default) and one to output data to Cassandra using this new mechanism. See that example in the latest release for details. + As of 0.7, there is be a basic mechanism included in Cassandra for outputting data to Cassandra. The `contrib/word_count` example in 0.7 contains two reducers - one for outputting data to the filesystem (default) and one to output data to Cassandra using this new mechanism. See that example in the latest release for details. Hadoop Streaming @@ -62, +63 @@ Anchor(Pig) == Pig == - Cassandra 0.6+ also adds support for [[http://hadoop.apache.org/pig/|Pig]] with its own implementation of [[http://hadoop.apache.org/pig/docs/r0.7.0/api/org/apache/pig/LoadFunc.html|LoadFunc]]. This allows Pig queries to be run against data stored in Cassandra. For an example of this, see the contrib/pig example in 0.6 and later. + Cassandra 0.6+ also adds support for [[http://pig.apache.org|Pig]] with its own implementation of [[http://pig.apache.org/docs/r0.7.0/api/org/apache/pig/LoadFunc.html|LoadFunc]]. This allows Pig queries to be run against data stored in Cassandra. For an example of this, see the `contrib/pig` example in 0.6 and later. + + When running Pig with Cassandra + Hadoop on a cluster, be sure to follow the
[Cassandra Wiki] Trivial Update of HadoopSupport by jeremyhanna
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The HadoopSupport page has been changed by jeremyhanna. http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=21rev2=22 -- }}} Virtual Datacenter - One thing that many have asked about is whether Cassandra with Hadoop will be usable from a random access perspective. For example, you may need to use Cassandra for serving web latency requests. You may also need to run analytics over your data. In Cassandra 0.7+ there is the NetworkTopologyStrategy which allows you to customize your cluster's replication strategy by datacenter. What you can do with this is create a 'virtual datacenter' to separate nodes that serve data with high random-read performance from nodes that are meant to be used for analytics. You need to have a snitch configured with your topology and then according to the datacenters defined there (either explicitly or implicitly), you can indicate how many replicas you would like in each datacenter. You would install task trackers on nodes in your analytics section and make sure that a replica is written to that 'datacenter' in your NetworkTopologyStrategy configuration. The practical upshot of this is your analytics nodes always have current data and your high random-read performance nodes always serve data with predictable performance. + One thing that many have asked about is whether Cassandra with Hadoop will be usable from a random access perspective. For example, you may need to use Cassandra for serving web latency requests. You may also need to run analytics over your data. In Cassandra 0.7+ there is the !NetworkTopologyStrategy which allows you to customize your cluster's replication strategy by datacenter. What you can do with this is create a 'virtual datacenter' to separate nodes that serve data with high random-read performance from nodes that are meant to be used for analytics. You need to have a snitch configured with your topology and then according to the datacenters defined there (either explicitly or implicitly), you can indicate how many replicas you would like in each datacenter. You would install task trackers on nodes in your analytics section and make sure that a replica is written to that 'datacenter' in your !NetworkTopologyStrategy configuration. The practical upshot of this is your analytics nodes always have current data and your high random-read performance nodes always serve data with predictable performance. For an example of configuring Cassandra with Hadoop in the cloud, see the [[http://github.com/digitalreasoning/PyStratus|PyStratus]] project on Github.
[Cassandra Wiki] Update of HadoopSupport by BrandonWilliams
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The HadoopSupport page has been changed by BrandonWilliams. http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=20rev2=21 -- Output To Cassandra - As of 0.7, there is be a basic mechanism included in Cassandra for outputting data to Cassandra. The `contrib/word_count` example in 0.7 contains two reducers - one for outputting data to the filesystem (default) and one to output data to Cassandra using this new mechanism. See that example in the latest release for details. + As of 0.7, there is be a basic mechanism included in Cassandra for outputting data to Cassandra. The `contrib/word_count` example in 0.7 contains two reducers - one for outputting data to the filesystem and one to output data to Cassandra (default) using this new mechanism. See that example in the latest release for details. Hadoop Streaming
[Cassandra Wiki] Update of HadoopSupport by jeremyhanna
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The HadoopSupport page has been changed by jeremyhanna. http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=22rev2=23 -- == Contents == * [[#overview|Overview]] - * [[#MapReduce|MapReduce Support]] + * [[#MapReduce|MapReduce]] - * [[#Pig|Pig Support]] + * [[#Pig|Pig]] - * [[#Hive|Hive Support]] + * [[#Hive|Hive]] * [[#ClusterConfig|Cluster Configuration]] * [[#Support|Support]]
[Cassandra Wiki] Update of HadoopSupport by jeremyhanna
Dear Wiki user, You have subscribed to a wiki page or wiki category on Cassandra Wiki for change notification. The HadoopSupport page has been changed by jeremyhanna. The comment on this change is: Adding a support options link for hadoop as well.. http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=23rev2=24 -- == Support == Sometimes configuration and integration can get tricky. To get support for this functionality, start with the `contrib` examples in the source download of Cassandra. Make sure you are following instructions in the `README` file for that example. You can search the Cassandra user mailing list or post on there as it is very active. You can also ask in the #Cassandra irc channel on freenode for help. Other channels that might be of use are #hadoop, #hadoop-pig, and #hive. Those projects' mailing lists are also very active. - There are professional support options for Cassandra that can help you get everything working together. For more information, see ThirdPartySupport. + There are professional support options for Cassandra that can help you get everything working together. For more information, see ThirdPartySupport. There are also professional support options specifically for Hadoop. For more information on that, see Hadoop's third party support [[http://wiki.apache.org/hadoop/Support|wiki page]]. [[#Top|Top]]
[Cassandra Wiki] Trivial Update of HadoopSupport by EricGilmore
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Cassandra Wiki for
change notification.
The HadoopSupport page has been changed by EricGilmore.
http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=24rev2=25
--
Anchor(MapReduce)
== MapReduce ==
-
Input from Cassandra
Cassandra 0.6+ adds support for retrieving data from Cassandra. This is
based on implementations of
[[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/InputSplit.html|InputSplit]],
[[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/InputFormat.html|InputFormat]],
and
[[http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/RecordReader.html|RecordReader]]
so that Hadoop !MapReduce jobs can retrieve data from Cassandra. For an
example of how this works, see the contrib/word_count example in 0.6 or later.
Cassandra rows or row fragments (that is, pairs of key + `SortedMap` of
columns) are input to Map tasks for processing by your job, as specified by a
`SlicePredicate` that describes which columns to fetch from each row.
@@ -31, +30 @@
SlicePredicate predicate = new
SlicePredicate().setColumn_names(Arrays.asList(columnName.getBytes()));
ConfigHelper.setSlicePredicate(job.getConfiguration(), predicate);
}}}
-
As of 0.7, configuration for Hadoop no longer resides in your job's specific
storage-conf.xml. See the `README` in the `word_count` and `pig` contrib
modules for more details.
Output To Cassandra
-
- As of 0.7, there is be a basic mechanism included in Cassandra for outputting
data to Cassandra. The `contrib/word_count` example in 0.7 contains two
reducers - one for outputting data to the filesystem and one to output data to
Cassandra (default) using this new mechanism. See that example in the latest
release for details.
+ As of 0.7, there is a basic mechanism included in Cassandra for outputting
data to Cassandra. The `contrib/word_count` example in 0.7 contains two
reducers - one for outputting data to the filesystem and one to output data to
Cassandra (default) using this new mechanism. See that example in the latest
release for details.
Hadoop Streaming
-
As of 0.7, there is support for
[[http://hadoop.apache.org/common/docs/r0.20.0/streaming.html|Hadoop
Streaming]]. For examples on how to use Streaming with Cassandra, see the
contrib section of the Cassandra source. The relevant tickets are
[[https://issues.apache.org/jira/browse/CASSANDRA-1368|CASSANDRA-1368]] and
[[https://issues.apache.org/jira/browse/CASSANDRA-1497|CASSANDRA-1497]].
Some troubleshooting
-
Releases before 0.6.2/0.7 are affected by a small resource leak that may
cause jobs to fail (connections are not released properly, causing a resource
leak). Depending on your local setup you may hit this issue, and workaround it
by raising the limit of open file descriptors for the process (e.g. in
linux/bash using `ulimit -n 32000`). The error will be reported on the hadoop
job side as a thrift !TimedOutException.
If you are testing the integration against a single node and you obtain some
failures, this may be normal: you are probably overloading the single machine,
which may again result in timeout errors. You can workaround it by reducing
the number of concurrent tasks
@@ -57, +52 @@
{{{
ConfigHelper.setRangeBatchSize(job.getConfiguration(), 1000);
}}}
-
[[#Top|Top]]
Anchor(Pig)
@@ -66, +60 @@
Cassandra 0.6+ also adds support for [[http://pig.apache.org|Pig]] with its
own implementation of
[[http://pig.apache.org/docs/r0.7.0/api/org/apache/pig/LoadFunc.html|LoadFunc]].
This allows Pig queries to be run against data stored in Cassandra. For an
example of this, see the `contrib/pig` example in 0.6 and later.
When running Pig with Cassandra + Hadoop on a cluster, be sure to follow the
`README` notes in the `cassandra_src/contrib/pig` directory, the
[[#ClusterConfig|Cluster Configuration]] section on this page, and some
additional notes here:
+
* Set the `HADOOP_HOME` environment variable to `hadoop_dir`, e.g.
`/opt/hadoop` or `/etc/hadoop`
* Set the `PIG_CONF` environment variable to `hadoop_dir/conf`
* Set the `JAVA_HOME`
@@ -82, +77 @@
Anchor(ClusterConfig)
== Cluster Configuration ==
-
If you would like to configure a Cassandra cluster so that Hadoop may operate
over its data, it's best to overlay a Hadoop cluster over your Cassandra nodes.
You'll want to have a separate server for your Hadoop namenode/`JobTracker`.
Then install a Hadoop `TaskTracker` on each of your Cassandra nodes. That will
allow the `Jobtracker` to assign tasks to the Cassandra nodes that contain data
for those tasks. At least one node in your cluster will also need to be a
datanode. That's because
