date:20110801

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073477#comment-13073477
]

Sylvain Lebresne commented on CASSANDRA-2901:
-

Comments:
* PCI.Reducer.getCompactedRow unwraps NotifyingSSTableIterators, so their
close() function won't be called (as a side note, it doesn't seem like we ever
call close() on the SSTableIdentityIterator).
* The MergeTask executor has a bounded queue (and number of threads), so tasks
can be rejected. If we want submitters to block when the queue is full and all
threads are occupied, we need to reuse the trick of
DebuggableThreadPoolExecutor.
* Deserializer uses a queue of size 1 to queue up to 1 row while it deserialize
the next one. However, we already queue up rows in the MergeTask executor, so
it feels like it would be simple to use direct handoff here. It would make it
easier to reason about how many rows are in memory at any given time for
instance.
* More generally, the memory blow up is (potentially) much more than the 2x
(compared to mono-threaded) in the description of this ticket. I think that
right now we may have:
** 1 for the row being deserialized
** 1 for the row in the Deserialized queue
** nbAvailProcessor's for the row in the MergeTask executor queue (each
mergeTask can contain up to 'InMemoryCompactionLimit' worth of data)
** 1 for the row being merged
Note that if we really want to get to the (roughly) 2x like in the
description of this ticket, we need direct hand-off for both the Deserializer
queue *and* the merge executor. I would be fine queuing a few tasks in the
merge executor though if that can help with throughput, but I'm not even sure
it will.
* MergeTask calls removeDeleted and removeOldShards on the compacted cf, but it
is also called in the constructor of PreCompactedRow a little bit later (we
should probably remove the occurrence in PreCompactedRow as it's still
multi-threaded while in the MergeTask).
* In PCI.Reducer.getCompactedRow, in the case where inMemory == false, it seems
we use the SSTI even for rows that were already read by the Deserializer, we
should use the row instead to avoid deserializing twice.

Nitpick:
* In the CompactionIterable (and PCI), we create one
ComparatorIColumnIterator each time instead of having a private static final
one (as it is the case prior to this patch). Granted, we don't create
compaction tasks quickly enough that it would really matter much, but it seems
like a good habit to be nice with the GC :)
* This is due to this patch, but there is a race when updating the bytesRead,
such that a user could get a 0 bytesRead temporarily in the middle of a big
compaction (and bytesRead should probably be volatile since it won't be read
for the same thread that write it).

Allow taking advantage of multiple cores while compacting a single CF
-

Key: CASSANDRA-2901
URL: https://issues.apache.org/jira/browse/CASSANDRA-2901
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Priority: Minor
Fix For: 0.8.3

Attachments: 2901-v2.txt, 2901.patch

Moved from CASSANDRA-1876:
There are five stages: read, deserialize, merge, serialize, and write. We
probably want to continue doing read+deserialize and serialize+write
together, or you waste a lot copying to/from buffers.
So, what I would suggest is: one thread per input sstable doing read +
deserialize (a row at a time). A thread pool (one per core?) merging
corresponding rows from each input sstable. One thread doing serialize +
writing the output (this has to wait for the merge threads to complete
in-order, obviously). This should take us from being CPU bound on SSDs (since
only one core is compacting) to being I/O bound.
This will require roughly 2x the memory, to allow the reader threads to work
ahead of the merge stage. (I.e. for each input sstable you will have up to
one row in a queue waiting to be merged, and the reader thread working on the
next.) Seems quite reasonable on that front. You'll also want a small queue
size for the serialize-merged-rows executor.
Multithreaded compaction should be either on or off. It doesn't make sense to
try to do things halfway (by doing the reads with a
threadpool whose size you can grow/shrink, for instance): we still have
compaction threads tuned to low priority, by default, so the impact on the
rest of the system won't be very different. Nor do we expect to have so many
input sstables that we lose a lot in context switching between reader threads.
IMO it's acceptable to punt completely on rows that are larger than memory,
and fall back to the old non-parallel code

[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073478#comment-13073478
 ] 

Sylvain Lebresne commented on CASSANDRA-2973:
-

Also, what is the story of that cluster. Is it a straight up 0.8 cluster, or 
was it upgraded from 0.7 ?

 fatal errrors after nodetool cleanup
 

 Key: CASSANDRA-2973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
Assignee: Sylvain Lebresne

 after adding nodes to cluster  running cleanup I get scaring exceptions in 
 log:
 2011-07-30 00:00:05:506 CEST ERROR 
 [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal 
 exception in thread Thread[ReadStage:2335,5,main]
 java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 
 4394 but 60165 requested
 at 
 org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146)
 at org.apache.cassandra.db.Table.getRow(Table.java:385)
 at 
 org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
 at 
 org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69)
 at 
 org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
  Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 
 but 60165 requested
 at 
 org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117)
 at 
 org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389)
 at 
 org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
 at 
 org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194)
 at 
 org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83)
 at 
 org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73)
 ... 14 more
 exceptions disappeared after running scrub

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2981) Provide Hadoop read access to Counter Columns.

2011-08-01 Thread Aaron Morton (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Morton updated CASSANDRA-2981:


Attachment: 0001-2981-hadoop-counters-input.patch

0001-2981-hadoop-counters-input.patch modifies the CFRR to turn CounterColumns 
returned through the thrift API into o.a.c.db.Column instances. 

Could not use the CounterColumn as the CounterContext needs to read the node 
ID, and this requires the StorageService to be running and access to 
cassandra.yaml.

It's not great, but the full CounterColumn should not be needed as Hadoop is 
read only access. Let me know it's too hacky.

Also added another test to the hadoop_word_count example that sums the counter 
columns in a row.  

 Provide Hadoop read access to Counter Columns.
 --

 Key: CASSANDRA-2981
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2981
 Project: Cassandra
  Issue Type: Improvement
  Components: Hadoop
Affects Versions: 0.8.2
Reporter: Aaron Morton
Assignee: Aaron Morton
Priority: Minor
 Attachments: 0001-2981-hadoop-counters-input.patch


 o.a.c.Hadoop.ColumnFamilyRecordReader does not test for counter columns, 
 which are different objects in the ColumnOrSuperColumn struct. Currently it 
 raises an error as it thinks it's a super column 
 {code:java}
 2011-07-26 17:23:34,376 ERROR CliDriver (SessionState.java:printError(343)) - 
 Failed with exception java.io.IOException:java.lang.NullPointerException
 java.io.IOException: java.lang.NullPointerException
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1114)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
 Caused by: java.lang.NullPointerException
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftifySuper(ColumnFamilyRecordReader.java:303)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftify(ColumnFamilyRecordReader.java:297)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:288)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
   at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
   at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
   at 
 org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:153)
   at 
 org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:111)
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
   ... 10 more
 {code}
 My plan is to return o.a.c.db.CounterColumn objects just like the 
 o.a.c.db.Column and SuperColumn that are returned.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2829) memtable with no post-flush activity can leave commitlog permanently dirty


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2829:


Priority: Critical  (was: Major)

 memtable with no post-flush activity can leave commitlog permanently dirty 
 ---

 Key: CASSANDRA-2829
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2829
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Aaron Morton
Assignee: Jonathan Ellis
Priority: Critical
 Fix For: 0.8.3

 Attachments: 0001-2829-unit-test-v08.patch, 
 0001-2829-unit-test.patch, 0002-2829-v08.patch, 0002-2829.patch, 2829.patch


 Only dirty Memtables are flushed, and so only dirty memtables are used to 
 discard obsolete commit log segments. This can result it log segments not 
 been deleted even though the data has been flushed.  
 Was using a 3 node 0.7.6-2 AWS cluster (DataStax AMI's) with pre 0.7 data 
 loaded and a running application working against the cluster. Did a rolling 
 restart and then kicked off a repair, one node filled up the commit log 
 volume with 7GB+ of log data, there was about 20 hours of log files. 
 {noformat}
 $ sudo ls -lah commitlog/
 total 6.9G
 drwx-- 2 cassandra cassandra  12K 2011-06-24 20:38 .
 drwxr-xr-x 3 cassandra cassandra 4.0K 2011-06-25 01:47 ..
 -rw--- 1 cassandra cassandra 129M 2011-06-24 01:08 
 CommitLog-1308876643288.log
 -rw--- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308876643288.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 01:36 
 CommitLog-1308877711517.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308877711517.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 02:20 
 CommitLog-1308879395824.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308879395824.log.header
 ...
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 20:38 
 CommitLog-1308946745380.log
 -rw-r--r-- 1 cassandra cassandra   36 2011-06-24 20:47 
 CommitLog-1308946745380.log.header
 -rw-r--r-- 1 cassandra cassandra 112M 2011-06-24 20:54 
 CommitLog-1308947888397.log
 -rw-r--r-- 1 cassandra cassandra   44 2011-06-24 20:47 
 CommitLog-1308947888397.log.header
 {noformat}
 The user KS has 2 CF's with 60 minute flush times. System KS had the default 
 settings which is 24 hours. Will create another ticket see if these can be 
 reduced or if it's something users should do, in this case it would not have 
 mattered. 
 I grabbed the log headers and used the tool in CASSANDRA-2828 and most of the 
 segments had the system CF's marked as dirty.
 {noformat}
 $ bin/logtool dirty /tmp/logs/commitlog/
 Not connected to a server, Keyspace and Column Family names are not available.
 /tmp/logs/commitlog/CommitLog-1308876643288.log.header
 Keyspace Unknown:
   Cf id 0: 444
 /tmp/logs/commitlog/CommitLog-1308877711517.log.header
 Keyspace Unknown:
   Cf id 1: 68848763
 ...
 /tmp/logs/commitlog/CommitLog-1308944451460.log.header
 Keyspace Unknown:
   Cf id 1: 61074
 /tmp/logs/commitlog/CommitLog-1308945597471.log.header
 Keyspace Unknown:
   Cf id 1000: 43175492
   Cf id 1: 108483
 /tmp/logs/commitlog/CommitLog-1308946745380.log.header
 Keyspace Unknown:
   Cf id 1000: 239223
   Cf id 1: 172211
 /tmp/logs/commitlog/CommitLog-1308947888397.log.header
 Keyspace Unknown:
   Cf id 1001: 57595560
   Cf id 1: 816960
   Cf id 1000: 0
 {noformat}
 CF 0 is the Status / LocationInfo CF and 1 is the HintedHandof CF. I dont 
 have it now, but IIRC CFStats showed the LocationInfo CF with dirty ops. 
 I was able to repo a case where flushing the CF's did not mark the log 
 segments as obsolete (attached unit-test patch). Steps are:
 1. Write to cf1 and flush.
 2. Current log segment is marked as dirty at the CL position when the flush 
 started, CommitLog.discardCompletedSegmentsInternal()
 3. Do not write to cf1 again.
 4. Roll the log, my test does this manually. 
 5. Write to CF2 and flush.
 6. Only CF2 is flushed because it is the only dirty CF. 
 cfs.maybeSwitchMemtable() is not called for cf1 and so log segment 1 is still 
 marked as dirty from cf1.
 Step 5 is not essential, just matched what I thought was happening. I thought 
 SystemTable.updateToken() was called which does not flush, and this was the 
 last thing that happened.  
 The expired memtable thread created by Table uses the same cfs.forceFlush() 
 which is a no-op if the cf or it's secondary indexes are clean. 
 
 I think the same problem would exist in 0.8. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2829) memtable with no post-flush activity can leave commitlog permanently dirty


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2829:


Attachment: 2829.patch

Houston, we have a problem.

In 0.8, we have a much bigger problem related to the commit log. Turns out we 
don't even turnOn the isDirty flag on writes. This means that typically if we 
fill a segment (with write of different cfs), starts a new one, and flush (one 
cf, say cf1), the previous segment will be removed even though it may be full 
of dirty writes for cf != cf1.

Attaching a patch that fix this issue as well as the original issue of this 
ticket (as it is not really more complicated). It adds two unit test, one for 
each issue (both fails in current 0.8). Bumping the priority of this too. 

 memtable with no post-flush activity can leave commitlog permanently dirty 
 ---

 Key: CASSANDRA-2829
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2829
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Aaron Morton
Assignee: Jonathan Ellis
 Fix For: 0.8.3

 Attachments: 0001-2829-unit-test-v08.patch, 
 0001-2829-unit-test.patch, 0002-2829-v08.patch, 0002-2829.patch, 2829.patch


 Only dirty Memtables are flushed, and so only dirty memtables are used to 
 discard obsolete commit log segments. This can result it log segments not 
 been deleted even though the data has been flushed.  
 Was using a 3 node 0.7.6-2 AWS cluster (DataStax AMI's) with pre 0.7 data 
 loaded and a running application working against the cluster. Did a rolling 
 restart and then kicked off a repair, one node filled up the commit log 
 volume with 7GB+ of log data, there was about 20 hours of log files. 
 {noformat}
 $ sudo ls -lah commitlog/
 total 6.9G
 drwx-- 2 cassandra cassandra  12K 2011-06-24 20:38 .
 drwxr-xr-x 3 cassandra cassandra 4.0K 2011-06-25 01:47 ..
 -rw--- 1 cassandra cassandra 129M 2011-06-24 01:08 
 CommitLog-1308876643288.log
 -rw--- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308876643288.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 01:36 
 CommitLog-1308877711517.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308877711517.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 02:20 
 CommitLog-1308879395824.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308879395824.log.header
 ...
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 20:38 
 CommitLog-1308946745380.log
 -rw-r--r-- 1 cassandra cassandra   36 2011-06-24 20:47 
 CommitLog-1308946745380.log.header
 -rw-r--r-- 1 cassandra cassandra 112M 2011-06-24 20:54 
 CommitLog-1308947888397.log
 -rw-r--r-- 1 cassandra cassandra   44 2011-06-24 20:47 
 CommitLog-1308947888397.log.header
 {noformat}
 The user KS has 2 CF's with 60 minute flush times. System KS had the default 
 settings which is 24 hours. Will create another ticket see if these can be 
 reduced or if it's something users should do, in this case it would not have 
 mattered. 
 I grabbed the log headers and used the tool in CASSANDRA-2828 and most of the 
 segments had the system CF's marked as dirty.
 {noformat}
 $ bin/logtool dirty /tmp/logs/commitlog/
 Not connected to a server, Keyspace and Column Family names are not available.
 /tmp/logs/commitlog/CommitLog-1308876643288.log.header
 Keyspace Unknown:
   Cf id 0: 444
 /tmp/logs/commitlog/CommitLog-1308877711517.log.header
 Keyspace Unknown:
   Cf id 1: 68848763
 ...
 /tmp/logs/commitlog/CommitLog-1308944451460.log.header
 Keyspace Unknown:
   Cf id 1: 61074
 /tmp/logs/commitlog/CommitLog-1308945597471.log.header
 Keyspace Unknown:
   Cf id 1000: 43175492
   Cf id 1: 108483
 /tmp/logs/commitlog/CommitLog-1308946745380.log.header
 Keyspace Unknown:
   Cf id 1000: 239223
   Cf id 1: 172211
 /tmp/logs/commitlog/CommitLog-1308947888397.log.header
 Keyspace Unknown:
   Cf id 1001: 57595560
   Cf id 1: 816960
   Cf id 1000: 0
 {noformat}
 CF 0 is the Status / LocationInfo CF and 1 is the HintedHandof CF. I dont 
 have it now, but IIRC CFStats showed the LocationInfo CF with dirty ops. 
 I was able to repo a case where flushing the CF's did not mark the log 
 segments as obsolete (attached unit-test patch). Steps are:
 1. Write to cf1 and flush.
 2. Current log segment is marked as dirty at the CL position when the flush 
 started, CommitLog.discardCompletedSegmentsInternal()
 3. Do not write to cf1 again.
 4. Roll the log, my test does this manually. 
 5. Write to CF2 and flush.
 6. Only CF2 is flushed because it is the only dirty CF. 
 cfs.maybeSwitchMemtable() is not called for cf1 and so log segment 1 is still 
 marked as dirty from cf1.
 Step 5 is not essential, just matched

[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup

2011-08-01 Thread JIRA


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073500#comment-13073500
 ] 

Michał Kowalczuk commented on CASSANDRA-2973:
-

Hello, Wojciech is my colleague. This cluster was upgraded from 0.7. I'm not 
sure which CF was getting read errors, though.

And as far as I know, Wojciech will be back on 15th August, not October.

 fatal errrors after nodetool cleanup
 

 Key: CASSANDRA-2973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
Assignee: Sylvain Lebresne

 after adding nodes to cluster  running cleanup I get scaring exceptions in 
 log:
 2011-07-30 00:00:05:506 CEST ERROR 
 [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal 
 exception in thread Thread[ReadStage:2335,5,main]
 java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 
 4394 but 60165 requested
 at 
 org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146)
 at org.apache.cassandra.db.Table.getRow(Table.java:385)
 at 
 org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
 at 
 org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69)
 at 
 org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
  Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 
 but 60165 requested
 at 
 org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117)
 at 
 org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389)
 at 
 org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
 at 
 org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194)
 at 
 org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83)
 at 
 org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73)
 ... 14 more
 exceptions disappeared after running scrub

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2968) AssertionError during compaction of CF with counter columns

2011-08-01 Thread Andriy Denysenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073513#comment-13073513
 ] 

Andriy Denysenko commented on CASSANDRA-2968:
-

no, I was wrong, compaction is not the culprit

I ran nodetool flush and produced 3 new small sstables - strangely they are all 
numbered with odd numbers only 153 155 157 - but the point is a freshly made 
sstable with only 8 rows is already unreadable by compaction/scrub.

Maybe header in counter binary value is not written or what...


{
00cc71a4: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201208019,c,-9223372036854775808]],
00748b45: 
[[clicks,00012335e4a0b2c911e06bb62b336af7009f009f23d44780b2c911e0f03fd2ec87ffa6cd8bac6eb8a32d2fd3c2059ade464925188750b2c911e058b7809d76ff9a8b961aed94ef799a8b961aed94ef79,1312201220709,c,-9223372036854775808]],
010a4465: 
[[clicks,2335e4a0b2c911e06bb62b336af745ad45ad23d44780b2c911e0f03fd2ec87ff4bfd01fae30325188750b2c911e058b7809d76ff038f038f,1312201219737,c,-9223372036854775808]],
01319592: 
[[clicks,00012335e4a0b2c911e06bb62b336af70001000123d44780b2c911e0f03fd2ec87ffdf1636876c1c1bc2df1636876c1c1bc225188750b2c911e058b7809d76ff0876b1020dc02ac97f12b88d9f9df751,1312201215475,c,-9223372036854775808]],
01b9a53e: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201202338,c,-9223372036854775808]],
004dd84f: 
[[clicks,00012335e4a0b2c911e06bb62b336af70001000123d44780b2c911e0f03fd2ec87ff0017d15559fb2ad925188750b2c911e058b7809d76ff000173bf000173bf,1312201205636,c,-9223372036854775808]],
00d1d52f: 
[[clicks,00012335e4a0b2c911e06bb62b336af700020002,1312201210515,c,-9223372036854775808]],
01410d4a: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201220365,c,-9223372036854775808]]
}


 AssertionError during compaction of CF with counter columns
 ---

 Key: CASSANDRA-2968
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2968
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: CentOS release 5.6
Reporter: Taras Puchko
Assignee: Sylvain Lebresne
 Fix For: 0.8.3

 Attachments: AffiliateActivity-g-147-Data.db, 
 AffiliateActivity-g-147-Index.db, AffiliateActivity-g-195-Data.db, 
 AffiliateActivity-g-195-Index.db


 Having upgraded from 0.8.0 to 0.8.2 we ran nodetool compact and got
 Error occured during compaction
 java.util.concurrent.ExecutionException: java.lang.AssertionError
 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
 at 
 org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
 at 
 com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
 at 
 com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
 at 
 com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
 at 
 com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
 at

[jira] [Issue Comment Edited] (CASSANDRA-2968) AssertionError during compaction of CF with counter columns

2011-08-01 Thread Andriy Denysenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073513#comment-13073513
 ] 

Andriy Denysenko edited comment on CASSANDRA-2968 at 8/1/11 12:39 PM:
--

no, I was wrong, compaction is not the culprit

I ran nodetool flush and produced 3 new small sstables - strangely they are all 
numbered with odd numbers only 153 155 157 - but the point is a freshly made 
sstable with only 8 rows is already unreadable by compaction/scrub.

Maybe header in counter binary value is not written or what...

{code}
{
00cc71a4: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201208019,c,-9223372036854775808]],
00748b45: 
[[clicks,00012335e4a0b2c911e06bb62b336af7009f009f23d44780b2c911e0f03fd2ec87ffa6cd8bac6eb8a32d2fd3c2059ade464925188750b2c911e058b7809d76ff9a8b961aed94ef799a8b961aed94ef79,1312201220709,c,-9223372036854775808]],
010a4465: 
[[clicks,2335e4a0b2c911e06bb62b336af745ad45ad23d44780b2c911e0f03fd2ec87ff4bfd01fae30325188750b2c911e058b7809d76ff038f038f,1312201219737,c,-9223372036854775808]],
01319592: 
[[clicks,00012335e4a0b2c911e06bb62b336af70001000123d44780b2c911e0f03fd2ec87ffdf1636876c1c1bc2df1636876c1c1bc225188750b2c911e058b7809d76ff0876b1020dc02ac97f12b88d9f9df751,1312201215475,c,-9223372036854775808]],
01b9a53e: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201202338,c,-9223372036854775808]],
004dd84f: 
[[clicks,00012335e4a0b2c911e06bb62b336af70001000123d44780b2c911e0f03fd2ec87ff0017d15559fb2ad925188750b2c911e058b7809d76ff000173bf000173bf,1312201205636,c,-9223372036854775808]],
00d1d52f: 
[[clicks,00012335e4a0b2c911e06bb62b336af700020002,1312201210515,c,-9223372036854775808]],
01410d4a: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201220365,c,-9223372036854775808]]
}
{code}

  was (Author: andrden):
no, I was wrong, compaction is not the culprit

I ran nodetool flush and produced 3 new small sstables - strangely they are all 
numbered with odd numbers only 153 155 157 - but the point is a freshly made 
sstable with only 8 rows is already unreadable by compaction/scrub.

Maybe header in counter binary value is not written or what...


{
00cc71a4: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201208019,c,-9223372036854775808]],
00748b45: 
[[clicks,00012335e4a0b2c911e06bb62b336af7009f009f23d44780b2c911e0f03fd2ec87ffa6cd8bac6eb8a32d2fd3c2059ade464925188750b2c911e058b7809d76ff9a8b961aed94ef799a8b961aed94ef79,1312201220709,c,-9223372036854775808]],
010a4465: 
[[clicks,2335e4a0b2c911e06bb62b336af745ad45ad23d44780b2c911e0f03fd2ec87ff4bfd01fae30325188750b2c911e058b7809d76ff038f038f,1312201219737,c,-9223372036854775808]],
01319592: 
[[clicks,00012335e4a0b2c911e06bb62b336af70001000123d44780b2c911e0f03fd2ec87ffdf1636876c1c1bc2df1636876c1c1bc225188750b2c911e058b7809d76ff0876b1020dc02ac97f12b88d9f9df751,1312201215475,c,-9223372036854775808]],
01b9a53e: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201202338,c,-9223372036854775808]],
004dd84f: 
[[clicks,00012335e4a0b2c911e06bb62b336af70001000123d44780b2c911e0f03fd2ec87ff0017d15559fb2ad925188750b2c911e058b7809d76ff000173bf000173bf,1312201205636,c,-9223372036854775808]],
00d1d52f: 
[[clicks,00012335e4a0b2c911e06bb62b336af700020002,1312201210515,c,-9223372036854775808]],
01410d4a: 
[[clicks,00012335e4a0b2c911e06bb62b336af700010001,1312201220365,c,-9223372036854775808]]
}

  
 AssertionError during compaction of CF with counter columns
 ---

 Key: CASSANDRA-2968
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2968
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: CentOS release 5.6
Reporter: Taras Puchko
Assignee: Sylvain Lebresne
 Fix For: 0.8.3

 Attachments: AffiliateActivity-g-147-Data.db, 
 AffiliateActivity-g-147-Index.db, AffiliateActivity-g-195-Data.db, 
 AffiliateActivity-g-195-Index.db


 Having upgraded from 0.8.0 to 0.8.2 we

[Cassandra Wiki] Update of ClientExamples by DmitryAsadullin

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The ClientExamples page has been changed by DmitryAsadullin:
http://wiki.apache.org/cassandra/ClientExamples?action=diffrev1=1rev2=3

- %@ page contentType=text/html;charset=UTF-8%
- %@ page language=java import=java.util.* pageEncoding=EUC-KR%
- %
- String path = request.getContextPath();
- String basePath = 
request.getScheme()+://+request.getServerName()+:+request.getServerPort()+path+/;
- %
  
+ package ru.easy.imagebase.cassandraclientexample;
- !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01 Transitional//EN
- html
-   head
- base href=%=basePath%
- 
- titleMy JSP 'index.jsp' starting page/title
-   meta http-equiv=pragma content=no-cache
-   meta http-equiv=cache-control content=no-cache
-   meta http-equiv=expires content=0
-   meta http-equiv=keywords content=keyword1,keyword2,keyword3
-   meta http-equiv=description content=This is my page
-   !--
-   link rel=stylesheet type=text/css href=styles.css
-   --
-   /head
-   
-   body
- Struts2 : This is my JSP page. 한국말 사용 가능br
-   /body
- /html
  
+ import java.util.List;
+ import java.io.UnsupportedEncodingException;
+ 
+ import java.nio.ByteBuffer;
+ import java.util.Random;
+ import org.apache.cassandra.thrift.AuthenticationException;
+ import org.apache.cassandra.thrift.AuthorizationException;
+ import org.apache.cassandra.thrift.Cassandra;
+ import org.apache.cassandra.thrift.Column;
+ import org.apache.cassandra.thrift.ColumnOrSuperColumn;
+ import org.apache.cassandra.thrift.ColumnParent;
+ import org.apache.cassandra.thrift.ConsistencyLevel;
+ import org.apache.cassandra.thrift.InvalidRequestException;
+ import org.apache.cassandra.thrift.KeyRange;
+ import org.apache.cassandra.thrift.KeySlice;
+ import org.apache.cassandra.thrift.NotFoundException;
+ import org.apache.cassandra.thrift.SlicePredicate;
+ import org.apache.cassandra.thrift.SliceRange;
+ import org.apache.cassandra.thrift.TimedOutException;
+ import org.apache.cassandra.thrift.UnavailableException;
+ import org.apache.thrift.transport.TTransport;
+ import org.apache.thrift.transport.TFramedTransport;
+ import org.apache.thrift.transport.TSocket;
+ import org.apache.thrift.protocol.TProtocol;
+ import org.apache.thrift.protocol.TBinaryProtocol;
+ import org.apache.thrift.TException;
+ import org.apache.thrift.transport.TTransportException;
+ 
+ public class App {
+ 
+   public static void main(String[] args) throws TTransportException, 
UnsupportedEncodingException, InvalidRequestException, NotFoundException, 
UnavailableException, TimedOutException, TException, AuthenticationException, 
AuthorizationException {
+ 
+   TTransport tr = new TFramedTransport(new TSocket(localhost, 
9160));
+   TProtocol proto = new TBinaryProtocol(tr);
+ 
+   Cassandra.Client client = new Cassandra.Client(proto);
+   tr.open();
+   String keyspace = Keyspace1;
+   client.set_keyspace(keyspace);
+   //record id
+   String key_user_id = 1;
+   String columnFamily = Standard1;
+   // insert data
+   long timestamp = System.currentTimeMillis();
+   Random r = new Random(timestamp);
+   Column nameColumn = new 
Column(ByteBuffer.wrap(name.getBytes()));
+   nameColumn.setValue(Long.toHexString(r.nextLong()).getBytes());
+   nameColumn.setTimestamp(timestamp);
+ 
+   Column ageColumn = new 
Column(ByteBuffer.wrap(age.getBytes()));
+   ageColumn.setValue(Long.toHexString(r.nextLong()).getBytes());
+   ageColumn.setTimestamp(timestamp);
+ 
+   ColumnParent columnParent = new ColumnParent(columnFamily);
+   client.insert(ByteBuffer.wrap(key_user_id.getBytes()), 
columnParent,nameColumn,ConsistencyLevel.ALL) ;
+   client.insert(ByteBuffer.wrap(key_user_id.getBytes()), 
columnParent,ageColumn,ConsistencyLevel.ALL);
+ 
+   //Gets column by key
+   SlicePredicate predicate = new SlicePredicate();
+   predicate.setSlice_range(new SliceRange(ByteBuffer.wrap(new 
byte[0]), ByteBuffer.wrap(new byte[0]), false, 100));
+   ListColumnOrSuperColumn columnsByKey = 
client.get_slice(ByteBuffer.wrap(key_user_id.getBytes()), columnParent, 
predicate, ConsistencyLevel.ALL);
+   System.out.println(columnsByKey);
+ 
+   
+   //Get all keys
+   KeyRange keyRange = new KeyRange(100);
+   keyRange.setStart_key(new byte[0]);
+   keyRange.setEnd_key(new byte[0]);
+   ListKeySlice keySlices = 
client.get_range_slices(columnParent, predicate, keyRange, 
ConsistencyLevel.ONE);
+   System.out.println(keySlices.size());
+   System.out.println(keySlices);
+

[Cassandra Wiki] Update of ClientExamples by DmitryAsadullin

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The ClientExamples page has been changed by DmitryAsadullin:
http://wiki.apache.org/cassandra/ClientExamples?action=diffrev1=3rev2=4

- 
+ {{{#!java
  package ru.easy.imagebase.cassandraclientexample;
  
  import java.util.List;
@@ -80, +80 @@

tr.close();
}
  }
+ }}}

[jira] [Commented] (CASSANDRA-2829) memtable with no post-flush activity can leave commitlog permanently dirty


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073539#comment-13073539
 ] 

Jonathan Ellis commented on CASSANDRA-2829:
---

+1

 memtable with no post-flush activity can leave commitlog permanently dirty 
 ---

 Key: CASSANDRA-2829
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2829
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Aaron Morton
Assignee: Jonathan Ellis
Priority: Critical
 Fix For: 0.8.3

 Attachments: 0001-2829-unit-test-v08.patch, 
 0001-2829-unit-test.patch, 0002-2829-v08.patch, 0002-2829.patch, 2829.patch


 Only dirty Memtables are flushed, and so only dirty memtables are used to 
 discard obsolete commit log segments. This can result it log segments not 
 been deleted even though the data has been flushed.  
 Was using a 3 node 0.7.6-2 AWS cluster (DataStax AMI's) with pre 0.7 data 
 loaded and a running application working against the cluster. Did a rolling 
 restart and then kicked off a repair, one node filled up the commit log 
 volume with 7GB+ of log data, there was about 20 hours of log files. 
 {noformat}
 $ sudo ls -lah commitlog/
 total 6.9G
 drwx-- 2 cassandra cassandra  12K 2011-06-24 20:38 .
 drwxr-xr-x 3 cassandra cassandra 4.0K 2011-06-25 01:47 ..
 -rw--- 1 cassandra cassandra 129M 2011-06-24 01:08 
 CommitLog-1308876643288.log
 -rw--- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308876643288.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 01:36 
 CommitLog-1308877711517.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308877711517.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 02:20 
 CommitLog-1308879395824.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308879395824.log.header
 ...
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 20:38 
 CommitLog-1308946745380.log
 -rw-r--r-- 1 cassandra cassandra   36 2011-06-24 20:47 
 CommitLog-1308946745380.log.header
 -rw-r--r-- 1 cassandra cassandra 112M 2011-06-24 20:54 
 CommitLog-1308947888397.log
 -rw-r--r-- 1 cassandra cassandra   44 2011-06-24 20:47 
 CommitLog-1308947888397.log.header
 {noformat}
 The user KS has 2 CF's with 60 minute flush times. System KS had the default 
 settings which is 24 hours. Will create another ticket see if these can be 
 reduced or if it's something users should do, in this case it would not have 
 mattered. 
 I grabbed the log headers and used the tool in CASSANDRA-2828 and most of the 
 segments had the system CF's marked as dirty.
 {noformat}
 $ bin/logtool dirty /tmp/logs/commitlog/
 Not connected to a server, Keyspace and Column Family names are not available.
 /tmp/logs/commitlog/CommitLog-1308876643288.log.header
 Keyspace Unknown:
   Cf id 0: 444
 /tmp/logs/commitlog/CommitLog-1308877711517.log.header
 Keyspace Unknown:
   Cf id 1: 68848763
 ...
 /tmp/logs/commitlog/CommitLog-1308944451460.log.header
 Keyspace Unknown:
   Cf id 1: 61074
 /tmp/logs/commitlog/CommitLog-1308945597471.log.header
 Keyspace Unknown:
   Cf id 1000: 43175492
   Cf id 1: 108483
 /tmp/logs/commitlog/CommitLog-1308946745380.log.header
 Keyspace Unknown:
   Cf id 1000: 239223
   Cf id 1: 172211
 /tmp/logs/commitlog/CommitLog-1308947888397.log.header
 Keyspace Unknown:
   Cf id 1001: 57595560
   Cf id 1: 816960
   Cf id 1000: 0
 {noformat}
 CF 0 is the Status / LocationInfo CF and 1 is the HintedHandof CF. I dont 
 have it now, but IIRC CFStats showed the LocationInfo CF with dirty ops. 
 I was able to repo a case where flushing the CF's did not mark the log 
 segments as obsolete (attached unit-test patch). Steps are:
 1. Write to cf1 and flush.
 2. Current log segment is marked as dirty at the CL position when the flush 
 started, CommitLog.discardCompletedSegmentsInternal()
 3. Do not write to cf1 again.
 4. Roll the log, my test does this manually. 
 5. Write to CF2 and flush.
 6. Only CF2 is flushed because it is the only dirty CF. 
 cfs.maybeSwitchMemtable() is not called for cf1 and so log segment 1 is still 
 marked as dirty from cf1.
 Step 5 is not essential, just matched what I thought was happening. I thought 
 SystemTable.updateToken() was called which does not flush, and this was the 
 last thing that happened.  
 The expired memtable thread created by Table uses the same cfs.forceFlush() 
 which is a no-op if the cf or it's secondary indexes are clean. 
 
 I think the same problem would exist in 0.8. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CASSANDRA-2955) Hadoop cannot read CounterColumn


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis resolved CASSANDRA-2955.
---

   Resolution: Duplicate
Fix Version/s: (was: 0.8.3)

addressed in CASSANDRA-2981

 Hadoop cannot read CounterColumn
 

 Key: CASSANDRA-2955
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2955
 Project: Cassandra
  Issue Type: New Feature
  Components: Hadoop
Affects Versions: 0.8.0
Reporter: Jacek Gerbszt
Priority: Minor
  Labels: lhf

 Accessing CounterColumn from Hadoop ends up with an exception:
 {noformat}
 java.lang.NullPointerException
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftifySuper(ColumnFamilyRecordReader.java:303)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftify(ColumnFamilyRecordReader.java:297)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:288)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
   at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:137)
   at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:132)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
   at org.apache.cassandra.hadoop.pig.CassandraStorage.getNext(Unknown 
 Source)
   at 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:187)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:456)
   at 
 org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
   at 
 org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
 {noformat}
 If you look into ColumnFamilyRecordRerader, you will quickly realize that it 
 doesn't know about CounterColumn's existence:
 {code:title=ColumnFamilyRecordRerader.java}
 private IColumn unthriftify(ColumnOrSuperColumn cosc)
 {
 if (cosc.column == null)
 return unthriftifySuper(cosc.super_column);
 return unthriftifySimple(cosc.column);
 }
 {code}
 It uses only column and super_column fields from ColumnOrSuperColumn class, 
 while this class keeps CounterColumn in one of two additional fields - 
 counter_column or counter_super_column:
 {code:title=ColumnOrSuperColumn.java}
 public class ColumnOrSuperColumn {
 public org.apache.cassandra.thrift.Column column;
 public org.apache.cassandra.thrift.SuperColumn super_column;
 public org.apache.cassandra.thrift.CounterColumn counter_column;
 public org.apache.cassandra.thrift.CounterSuperColumn 
 counter_super_column;
 }
 {code}
 Please add CounterColumn support to ColumnFamilyRecordReader class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1152782 - in /cassandra/branches/cassandra-0.8: ./ examples/hadoop_word_count/ examples/hadoop_word_count/bin/ examples/hadoop_word_count/src/ src/java/org/apache/cassandra/hadoop/

2011-08-01 Thread jbellis

Author: jbellis
Date: Mon Aug  1 14:05:34 2011
New Revision: 1152782

URL: http://svn.apache.org/viewvc?rev=1152782view=rev
Log:
add counter support to Hadoop InputFormat
patch by Aaron Morton; reviewed by jbellis for CASSANDRA-2981

Added:

cassandra/branches/cassandra-0.8/examples/hadoop_word_count/bin/word_count_counters

cassandra/branches/cassandra-0.8/examples/hadoop_word_count/src/WordCountCounters.java
Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt
cassandra/branches/cassandra-0.8/examples/hadoop_word_count/README.txt

cassandra/branches/cassandra-0.8/examples/hadoop_word_count/src/WordCountSetup.java

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1152782r1=1152781r2=1152782view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Mon Aug  1 14:05:34 2011
@@ -16,6 +16,7 @@
  * Flush memtables on shutdown when durable writes are disabled 
(CASSANDRA-2958)
  * improved POSIX compatibility of start scripts (CASsANDRA-2965)
+ * add counter support to Hadoop InputFormat (CASSANDRA-2981)
 
 
 0.8.2

Modified: cassandra/branches/cassandra-0.8/examples/hadoop_word_count/README.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/examples/hadoop_word_count/README.txt?rev=1152782r1=1152781r2=1152782view=diff
==
--- cassandra/branches/cassandra-0.8/examples/hadoop_word_count/README.txt 
(original)
+++ cassandra/branches/cassandra-0.8/examples/hadoop_word_count/README.txt Mon 
Aug  1 14:05:34 2011
@@ -1,5 +1,6 @@
 WordCount hadoop example: Inserts a bunch of words across multiple rows,
-and counts them, with RandomPartitioner.
+and counts them, with RandomPartitioner. The word_count_counters example sums
+the value of counter columns for a key.
 
 The scripts in bin/ assume you are running with cwd of contrib/word_count.
 
@@ -9,6 +10,7 @@ then run
 contrib/word_count$ ant
 contrib/word_count$ bin/word_count_setup
 contrib/word_count$ bin/word_count
+contrib/word_count$ bin/word_count_counters
 
 In order to view the results in Cassandra, one can use bin/cassandra-cli and
 perform the following operations:
@@ -25,5 +27,8 @@ in the 'wordcount' keyspace.  'cassandra
 
 Read the code in src/ for more details.
 
+The word_count_counters example sums the counter columns for a row. The output
+is written to a text file in /tmp/word_count_counters.
+
 *If you want to point wordcount at a real cluster, modify the seed
 and listenaddress settings accordingly.

Added: 
cassandra/branches/cassandra-0.8/examples/hadoop_word_count/bin/word_count_counters
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/examples/hadoop_word_count/bin/word_count_counters?rev=1152782view=auto
==
--- 
cassandra/branches/cassandra-0.8/examples/hadoop_word_count/bin/word_count_counters
 (added)
+++ 
cassandra/branches/cassandra-0.8/examples/hadoop_word_count/bin/word_count_counters
 Mon Aug  1 14:05:34 2011
@@ -0,0 +1,58 @@
+#!/bin/sh
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# License); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cwd=`dirname $0`
+
+# Cassandra class files.
+if [ ! -d $cwd/../../../build/classes/main ]; then
+echo Unable to locate cassandra class files 2
+exit 1
+fi
+
+# word_count Jar.
+if [ ! -e $cwd/../build/word_count.jar ]; then
+echo Unable to locate word_count jar 2
+exit 1
+fi
+
+CLASSPATH=$CLASSPATH:$cwd/../build/word_count.jar
+CLASSPATH=$CLASSPATH:$cwd/../../../build/classes/main
+CLASSPATH=$CLASSPATH:$cwd/../../../build/classes/thrift
+for jar in $cwd/../build/lib/jars/*.jar; do
+CLASSPATH=$CLASSPATH:$jar
+done
+for jar in $cwd/../../../lib/*.jar; do
+CLASSPATH=$CLASSPATH:$jar
+done
+for jar in $cwd/../../../build/lib/jars/*.jar; do
+CLASSPATH=$CLASSPATH:$jar
+done
+
+if [ -x $JAVA_HOME/bin/java ]; then
+JAVA=$JAVA_HOME/bin/java
+else
+

svn commit: r1152793 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/db/commitlog/CommitLog.java src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java test

Author: slebresne
Date: Mon Aug  1 14:22:50 2011
New Revision: 1152793

URL: http://svn.apache.org/viewvc?rev=1152793view=rev
Log:
fix bug where dirty commit logs were removed (and avoid keeping segment with no 
post-flush activity permanently dirty)
patch by slebresne; reviewed by jbellis for CASSANDRA-2829

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLog.java

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java

cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/db/CommitLogTest.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1152793r1=1152792r2=1152793view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Mon Aug  1 14:22:50 2011
@@ -17,6 +17,8 @@
(CASSANDRA-2958)
  * improved POSIX compatibility of start scripts (CASsANDRA-2965)
  * add counter support to Hadoop InputFormat (CASSANDRA-2981)
+ * fix bug where dirty commit logs were removed (and avoid keeping segment
+   with no post-flush activity permanently dirty) (CASSANDRA-2829)
 
 
 0.8.2

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLog.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLog.java?rev=1152793r1=1152792r2=1152793view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLog.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLog.java
 Mon Aug  1 14:22:50 2011
@@ -385,6 +385,12 @@ public class CommitLog
 }
 }
 
+// for tests mainly
+public int segmentsCount()
+{
+return segments.size();
+}
+
 /*
  * Adds the specified row to the commit log. This method will reset the
  * file offset to what it is before the start of the operation in case
@@ -450,30 +456,36 @@ public class CommitLog
 CommitLogSegment segment = iter.next();
 if (segment.id == context.segment)
 {
-// we can't just mark the segment where the flush happened 
clean,
-// since there may have been writes to it between when the 
flush
-// started and when it finished.
-segment.turnOn(id);
+// Only unmark this segment if there were not write since the
+// ReplayPosition was grabbed.
+segment.turnOffIfNotWritten(id, context.position);
+maybeDiscardSegment(segment, iter);
 break;
 }
 
 segment.turnOff(id);
-if (segment.isSafeToDelete()  iter.hasNext())
-{
-logger.info(Discarding obsolete commit log: + segment);
-segment.close();
-DeletionService.executeDelete(segment.getPath());
-// usually this will be the first (remaining) segment, but not 
always, if segment A contains
-// writes to a CF that is unflushed but is followed by segment 
B whose CFs are all flushed.
-iter.remove();
-}
-else
-{
-if (logger.isDebugEnabled())
-logger.debug(Not safe to delete commit log  + segment + 
; dirty is  + segment.dirtyString() + ; hasNext:  + iter.hasNext());
-}
+maybeDiscardSegment(segment, iter);
 }
 }
+
+private void maybeDiscardSegment(CommitLogSegment segment, 
IteratorCommitLogSegment iter)
+{
+if (segment.isSafeToDelete()  iter.hasNext())
+{
+logger.info(Discarding obsolete commit log: + segment);
+segment.close();
+DeletionService.executeDelete(segment.getPath());
+// usually this will be the first (remaining) segment, but not 
always, if segment A contains
+// writes to a CF that is unflushed but is followed by segment B 
whose CFs are all flushed.
+iter.remove();
+}
+else
+{
+if (logger.isDebugEnabled())
+logger.debug(Not safe to delete commit log  + segment + ; 
dirty is  + segment.dirtyString() + ; hasNext:  + iter.hasNext());
+}
+}
+
 
 void sync() throws IOException
 {

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java?rev=1152793r1=1152792r2=1152793view=diff

svn commit: r1152795 - in /cassandra/branches/cassandra-0.8: ./ src/java/org/apache/cassandra/db/ test/unit/org/apache/cassandra/db/

Author: slebresne
Date: Mon Aug  1 14:26:24 2011
New Revision: 1152795

URL: http://svn.apache.org/viewvc?rev=1152795view=rev
Log:
don't throw exception on batch of counter super columns
patch by slebresne; reviewed by soverton for CASSANDRA-2949

Added:

cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/db/CounterMutationTest.java
Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamily.java

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/CounterMutation.java

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/RowMutation.java

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/SuperColumn.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1152795r1=1152794r2=1152795view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Mon Aug  1 14:26:24 2011
@@ -19,6 +19,8 @@
  * add counter support to Hadoop InputFormat (CASSANDRA-2981)
  * fix bug where dirty commit logs were removed (and avoid keeping segment
with no post-flush activity permanently dirty) (CASSANDRA-2829)
+ * fix throwing exception with batch mutation of counter super columns
+   (CASSANDRA-2949)
 
 
 0.8.2

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamily.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamily.java?rev=1152795r1=1152794r2=1152795view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamily.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamily.java
 Mon Aug  1 14:26:24 2011
@@ -419,6 +419,43 @@ public class ColumnFamily implements ICo
 return getColumnCount();
 }
 
+public void retainAll(ColumnFamily cf)
+{
+IteratorIColumn iter = iterator();
+IteratorIColumn toRetain = cf.iterator();
+IColumn current = iter.hasNext() ? iter.next() : null;
+IColumn retain = toRetain.hasNext() ? toRetain.next() : null;
+AbstractType comparator = getComparator();
+while (current != null  retain != null)
+{
+int c = comparator.compare(current.name(), retain.name());
+if (c == 0)
+{
+if (isSuper())
+{
+assert current instanceof SuperColumn  retain instanceof 
SuperColumn;
+((SuperColumn)current).retainAll((SuperColumn)retain);
+}
+current = iter.hasNext() ? iter.next() : null;
+retain = toRetain.hasNext() ? toRetain.next() : null;
+}
+else if (c  0)
+{
+iter.remove();
+current = iter.hasNext() ? iter.next() : null;
+}
+else // c  0
+{
+retain = toRetain.hasNext() ? toRetain.next() : null;
+}
+}
+while (current != null)
+{
+iter.remove();
+current = iter.hasNext() ? iter.next() : null;
+}
+}
+
 public IteratorIColumn iterator()
 {
 return columns.values().iterator();

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/CounterMutation.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/CounterMutation.java?rev=1152795r1=1152794r2=1152795view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/CounterMutation.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/CounterMutation.java
 Mon Aug  1 14:26:24 2011
@@ -117,37 +117,18 @@ public class CounterMutation implements 
 continue;
 
 row = mergeOldShards(readCommand.table, row);
-replicationMutation.add(row.cf);
+ColumnFamily cf = row.cf;
+if (cf.isSuper())
+cf.retainAll(rowMutation.getColumnFamily(cf.metadata().cfId));
+replicationMutation.add(cf);
 }
 return replicationMutation;
 }
 
 private void addReadCommandFromColumnFamily(String table, ByteBuffer key, 
ColumnFamily columnFamily, ListReadCommand commands)
 {
-// CF type: regular
-if (!columnFamily.isSuper())
-{
-QueryPath queryPath = new 
QueryPath(columnFamily.metadata().cfName);
-commands.add(new SliceByNamesReadCommand(table, key, queryPath, 
columnFamily.getColumnNames()));
-

svn commit: r1152808 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/db/ test/unit/org/apache/cassandra/db/

Author: slebresne
Date: Mon Aug  1 14:47:04 2011
New Revision: 1152808

URL: http://svn.apache.org/viewvc?rev=1152808view=rev
Log:
Merge from 0.8

Modified:
cassandra/trunk/   (props changed)
cassandra/trunk/CHANGES.txt
cassandra/trunk/contrib/   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
   (props changed)

cassandra/trunk/src/java/org/apache/cassandra/db/AbstractColumnContainer.java
cassandra/trunk/src/java/org/apache/cassandra/db/CounterMutation.java
cassandra/trunk/src/java/org/apache/cassandra/db/RowMutation.java
cassandra/trunk/src/java/org/apache/cassandra/db/SuperColumn.java
cassandra/trunk/test/unit/org/apache/cassandra/db/CounterMutationTest.java

Propchange: cassandra/trunk/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 14:47:04 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291
 /cassandra/branches/cassandra-0.7:1026516-1151306
 /cassandra/branches/cassandra-0.7.0:1053690-1055654
-/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1152110,1152265,1152793
+/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1152110,1152265,1152793,1152795
 /cassandra/branches/cassandra-0.8.0:1125021-1130369
 /cassandra/branches/cassandra-0.8.1:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689

Modified: cassandra/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1152808r1=1152807r2=1152808view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Mon Aug  1 14:47:04 2011
@@ -41,6 +41,8 @@
  * speedup bytes to hex conversions dramatically (CASSANDRA-2850)
  * fix bug where dirty commit logs were removed (and avoid keeping segment
with no post-flush activity permanently dirty) (CASSANDRA-2829)
+ * fix throwing exception with batch mutation of counter super columns
+   (CASSANDRA-2949)
 
 
 0.8.2

Propchange: cassandra/trunk/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 14:47:04 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
 /cassandra/branches/cassandra-0.7/contrib:1026516-1151306
 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
-/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1152110,1152265,1152793
+/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1152110,1152265,1152793,1152795
 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369
 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 14:47:04 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1151306
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654
-/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1152110,1152265,1152793
+/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1152110,1152265,1152793,1152795
 
/cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369
 
/cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018
 
/cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 14:47:04 2011
@@

[Cassandra Wiki] Update of HadoopSupport by jeremyhanna

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The HadoopSupport page has been changed by jeremyhanna:
http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=37rev2=38

Comment:
Updated cluster config section.  Took out single datanode thought as 
intermediate results need more than that realistically.  Added bit about Brisk.

  Anchor(ClusterConfig)
  
  == Cluster Configuration ==
- If you would like to configure a Cassandra cluster so that Hadoop may operate 
over its data, it's best to overlay a Hadoop cluster over your Cassandra nodes. 
 You'll want to have a separate server for your Hadoop namenode/`JobTracker`.  
Then install a Hadoop `TaskTracker` on each of your Cassandra nodes.  That will 
allow the `Jobtracker` to assign tasks to the Cassandra nodes that contain data 
for those tasks.  At least one node in your cluster will also need to be a 
datanode.  That's because Hadoop uses HDFS to store information like jar 
dependencies for your job, static data (like stop words for a word count), and 
things like that - it's the distributed cache.  It's a very small amount of 
data but the Hadoop cluster needs it to run properly.
+ 
+ The simplest way to configure your cluster to run Cassandra with Hadoop is to 
use Brisk, the open-source packaging of Cassandra with Hadoop.  That will start 
the `JobTracker` and `TaskTracker` processes for you.  It also uses CFS, an 
HDFS compatible distributed filesystem built on Cassandra that removes the need 
for a Hadoop `NameNode` and `DataNode` processes.  For details, see the Brisk 
[[http://www.datastax.com/docs/0.8/brisk/index|documentation]] and 
[[http://github.com/riptano/brisk|code]]
+ 
+ Otherwise, if you would like to configure a Cassandra cluster yourself so 
that Hadoop may operate over its data, it's best to overlay a Hadoop cluster 
over your Cassandra nodes.  You'll want to have a separate server for your 
Hadoop `NameNode/`JobTracker`.  Then install a Hadoop `TaskTracker` on each of 
your Cassandra nodes.  That will allow the `JobTracker` to assign tasks to the 
Cassandra nodes that contain data for those tasks.  Also install a Hadoop 
`DataNode` on each Cassandra node.  Hadoop requires a distributed filesystem 
for copying dependency jars, static data, and intermediate results to be stored.
  
  The nice thing about having a `TaskTracker` on every node is that you get 
data locality and your analytics engine scales with your data. You also never 
need to shuttle around your data once you've performed analytics on it - you 
simply output to Cassandra and you are able to access that data with high 
random-read performance.
  
@@ -79, +82 @@

  }}}
   Virtual Datacenter 
  One thing that many have asked about is whether Cassandra with Hadoop will be 
usable from a random access perspective. For example, you may need to use 
Cassandra for serving web latency requests. You may also need to run analytics 
over your data. In Cassandra 0.7+ there is the !NetworkTopologyStrategy which 
allows you to customize your cluster's replication strategy by datacenter. What 
you can do with this is create a 'virtual datacenter' to separate nodes that 
serve data with high random-read performance from nodes that are meant to be 
used for analytics. You need to have a snitch configured with your topology and 
then according to the datacenters defined there (either explicitly or 
implicitly), you can indicate how many replicas you would like in each 
datacenter. You would install task trackers on nodes in your analytics section 
and make sure that a replica is written to that 'datacenter' in your 
!NetworkTopologyStrategy configuration. The practical upshot of this is your 
analytics nodes always have current data and your high random-read performance 
nodes always serve data with predictable performance.
- 
- For an example of configuring Cassandra with Hadoop in the cloud, see the 
[[http://github.com/digitalreasoning/PyStratus|PyStratus]] project on Github.
  
  [[#Top|Top]]

[Cassandra Wiki] Trivial Update of HadoopSupport by jeremyhanna

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The HadoopSupport page has been changed by jeremyhanna:
http://wiki.apache.org/cassandra/HadoopSupport?action=diffrev1=38rev2=39

Comment:
fixed a typo.

  
  The simplest way to configure your cluster to run Cassandra with Hadoop is to 
use Brisk, the open-source packaging of Cassandra with Hadoop.  That will start 
the `JobTracker` and `TaskTracker` processes for you.  It also uses CFS, an 
HDFS compatible distributed filesystem built on Cassandra that removes the need 
for a Hadoop `NameNode` and `DataNode` processes.  For details, see the Brisk 
[[http://www.datastax.com/docs/0.8/brisk/index|documentation]] and 
[[http://github.com/riptano/brisk|code]]
  
- Otherwise, if you would like to configure a Cassandra cluster yourself so 
that Hadoop may operate over its data, it's best to overlay a Hadoop cluster 
over your Cassandra nodes.  You'll want to have a separate server for your 
Hadoop `NameNode/`JobTracker`.  Then install a Hadoop `TaskTracker` on each of 
your Cassandra nodes.  That will allow the `JobTracker` to assign tasks to the 
Cassandra nodes that contain data for those tasks.  Also install a Hadoop 
`DataNode` on each Cassandra node.  Hadoop requires a distributed filesystem 
for copying dependency jars, static data, and intermediate results to be stored.
+ Otherwise, if you would like to configure a Cassandra cluster yourself so 
that Hadoop may operate over its data, it's best to overlay a Hadoop cluster 
over your Cassandra nodes.  You'll want to have a separate server for your 
Hadoop `NameNode`/`JobTracker`.  Then install a Hadoop `TaskTracker` on each of 
your Cassandra nodes.  That will allow the `JobTracker` to assign tasks to the 
Cassandra nodes that contain data for those tasks.  Also install a Hadoop 
`DataNode` on each Cassandra node.  Hadoop requires a distributed filesystem 
for copying dependency jars, static data, and intermediate results to be stored.
  
  The nice thing about having a `TaskTracker` on every node is that you get 
data locality and your analytics engine scales with your data. You also never 
need to shuttle around your data once you've performed analytics on it - you 
simply output to Cassandra and you are able to access that data with high 
random-read performance.

[jira] [Commented] (CASSANDRA-2949) Batch mutation of counters in multiple supercolumns throws an exception during replication.


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073575#comment-13073575
 ] 

Hudson commented on CASSANDRA-2949:
---

Integrated in Cassandra-0.8 #248 (See 
[https://builds.apache.org/job/Cassandra-0.8/248/])
don't throw exception on batch of counter super columns
patch by slebresne; reviewed by soverton for CASSANDRA-2949

slebresne : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1152795
Files : 
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/RowMutation.java
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/SuperColumn.java
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamily.java
* 
/cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/db/CounterMutationTest.java
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/CounterMutation.java


 Batch mutation of counters in multiple supercolumns throws an exception 
 during replication.
 ---

 Key: CASSANDRA-2949
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2949
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.2
Reporter: Sam Overton
Assignee: Sylvain Lebresne
Priority: Critical
  Labels: batch_mutate, counters, supercolumns
 Fix For: 0.8.3

 Attachments: 2949.patch, 2949_v2.patch


 Steps to reproduce:
 * Perform a batch mutation of more than one counter in more than one 
 super-column in the same column-family.
 * The following exception is thrown during replication:
 DEBUG [MutationStage:63] 2011-07-26 17:05:12,653 
 CounterMutationVerbHandler.java (line 52) Applying forwarded 
 CounterMutation(RowMutation(keyspace='ks1', key='4ae71336e44bf9bf', 
 modifications=[ColumnFamily(cf1 [SuperColumn(302c7375706572636f6c30 
 [302c636f6c30:false:8@1311696312648,]),SuperColumn(302c7375706572636f6c31 
 [302c636f6c30:false:8@1311696312648,]),])]), QUORUM)
 DEBUG [MutationStage:63] 2011-07-26 17:05:12,653 StorageProxy.java (line 432) 
 insert writing local  replicate CounterMutation(RowMutation(keyspace='ks1', 
 key='4ae71336e44bf9bf', modifications=[cf1]), QUORUM)
 DEBUG [MutationStage:63] 2011-07-26 17:05:12,654 Table.java (line 398) 
 applying mutation of row 4ae71336e44bf9bf
 ERROR [ReplicateOnWriteStage:125] 2011-07-26 17:05:12,655 
 AbstractCassandraDaemon.java (line 139) Fatal exception in thread 
 Thread[ReplicateOnWriteStage:125,5,main]
 java.lang.RuntimeException: java.lang.IllegalArgumentException: ColumnFamily 
 ColumnFamily(cf1 [SuperColumn(302c7375706572636f6c31 
 [302c636f6c30:false:[{cad93dc0-b7a0-11e0--123f813dd5df, 3, 
 3}*]@1311696312648!-9223372036854775808,]),]) already has modifications in 
 this mutation: ColumnFamily(cf1 [SuperColumn(302c7375706572636f6c30 
 [302c636f6c30:false:[{cad93dc0-b7a0-11e0--123f813dd5df, 3, 
 3}*]@1311696312648!-9223372036854775808,]),])
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:636)
 Caused by: java.lang.IllegalArgumentException: ColumnFamily ColumnFamily(cf1 
 [SuperColumn(302c7375706572636f6c31 
 [302c636f6c30:false:[{cad93dc0-b7a0-11e0--123f813dd5df, 3, 
 3}*]@1311696312648!-9223372036854775808,]),]) already has modifications in 
 this mutation: ColumnFamily(cf1 [SuperColumn(302c7375706572636f6c30 
 [302c636f6c30:false:[{cad93dc0-b7a0-11e0--123f813dd5df, 3, 
 3}*]@1311696312648!-9223372036854775808,]),])
 at org.apache.cassandra.db.RowMutation.add(RowMutation.java:123)
 at 
 org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:120)
 at 
 org.apache.cassandra.service.StorageProxy$5$1.runMayThrow(StorageProxy.java:455)
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
 ... 3 more

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2981) Provide Hadoop read access to Counter Columns.


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073573#comment-13073573
 ] 

Hudson commented on CASSANDRA-2981:
---

Integrated in Cassandra-0.8 #248 (See 
[https://builds.apache.org/job/Cassandra-0.8/248/])
add counter support to Hadoop InputFormat
patch by Aaron Morton; reviewed by jbellis for CASSANDRA-2981

jbellis : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1152782
Files : 
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java
* 
/cassandra/branches/cassandra-0.8/examples/hadoop_word_count/bin/word_count_counters
* 
/cassandra/branches/cassandra-0.8/examples/hadoop_word_count/src/WordCountCounters.java
* /cassandra/branches/cassandra-0.8/examples/hadoop_word_count/README.txt
* 
/cassandra/branches/cassandra-0.8/examples/hadoop_word_count/src/WordCountSetup.java


 Provide Hadoop read access to Counter Columns.
 --

 Key: CASSANDRA-2981
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2981
 Project: Cassandra
  Issue Type: Improvement
  Components: Hadoop
Affects Versions: 0.8.2
Reporter: Aaron Morton
Assignee: Aaron Morton
Priority: Minor
 Fix For: 0.8.3

 Attachments: 0001-2981-hadoop-counters-input.patch


 o.a.c.Hadoop.ColumnFamilyRecordReader does not test for counter columns, 
 which are different objects in the ColumnOrSuperColumn struct. Currently it 
 raises an error as it thinks it's a super column 
 {code:java}
 2011-07-26 17:23:34,376 ERROR CliDriver (SessionState.java:printError(343)) - 
 Failed with exception java.io.IOException:java.lang.NullPointerException
 java.io.IOException: java.lang.NullPointerException
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
   at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
   at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1114)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
 Caused by: java.lang.NullPointerException
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftifySuper(ColumnFamilyRecordReader.java:303)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftify(ColumnFamilyRecordReader.java:297)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:288)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
   at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
   at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
   at 
 org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
   at 
 org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:153)
   at 
 org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:111)
   at 
 org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
   ... 10 more
 {code}
 My plan is to return o.a.c.db.CounterColumn objects just like the 
 o.a.c.db.Column and SuperColumn that are returned.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2829) memtable with no post-flush activity can leave commitlog permanently dirty


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073574#comment-13073574
 ] 

Hudson commented on CASSANDRA-2829:
---

Integrated in Cassandra-0.8 #248 (See 
[https://builds.apache.org/job/Cassandra-0.8/248/])
fix bug where dirty commit logs were removed (and avoid keeping segment 
with no post-flush activity permanently dirty)
patch by slebresne; reviewed by jbellis for CASSANDRA-2829

slebresne : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1152793
Files : 
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
* 
/cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/db/CommitLogTest.java
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/commitlog/CommitLog.java


 memtable with no post-flush activity can leave commitlog permanently dirty 
 ---

 Key: CASSANDRA-2829
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2829
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Aaron Morton
Assignee: Sylvain Lebresne
Priority: Critical
 Fix For: 0.8.3

 Attachments: 0001-2829-unit-test-v08.patch, 
 0001-2829-unit-test.patch, 0002-2829-v08.patch, 0002-2829.patch, 2829.patch


 Only dirty Memtables are flushed, and so only dirty memtables are used to 
 discard obsolete commit log segments. This can result it log segments not 
 been deleted even though the data has been flushed.  
 Was using a 3 node 0.7.6-2 AWS cluster (DataStax AMI's) with pre 0.7 data 
 loaded and a running application working against the cluster. Did a rolling 
 restart and then kicked off a repair, one node filled up the commit log 
 volume with 7GB+ of log data, there was about 20 hours of log files. 
 {noformat}
 $ sudo ls -lah commitlog/
 total 6.9G
 drwx-- 2 cassandra cassandra  12K 2011-06-24 20:38 .
 drwxr-xr-x 3 cassandra cassandra 4.0K 2011-06-25 01:47 ..
 -rw--- 1 cassandra cassandra 129M 2011-06-24 01:08 
 CommitLog-1308876643288.log
 -rw--- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308876643288.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 01:36 
 CommitLog-1308877711517.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308877711517.log.header
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 02:20 
 CommitLog-1308879395824.log
 -rw-r--r-- 1 cassandra cassandra   28 2011-06-24 20:47 
 CommitLog-1308879395824.log.header
 ...
 -rw-r--r-- 1 cassandra cassandra 129M 2011-06-24 20:38 
 CommitLog-1308946745380.log
 -rw-r--r-- 1 cassandra cassandra   36 2011-06-24 20:47 
 CommitLog-1308946745380.log.header
 -rw-r--r-- 1 cassandra cassandra 112M 2011-06-24 20:54 
 CommitLog-1308947888397.log
 -rw-r--r-- 1 cassandra cassandra   44 2011-06-24 20:47 
 CommitLog-1308947888397.log.header
 {noformat}
 The user KS has 2 CF's with 60 minute flush times. System KS had the default 
 settings which is 24 hours. Will create another ticket see if these can be 
 reduced or if it's something users should do, in this case it would not have 
 mattered. 
 I grabbed the log headers and used the tool in CASSANDRA-2828 and most of the 
 segments had the system CF's marked as dirty.
 {noformat}
 $ bin/logtool dirty /tmp/logs/commitlog/
 Not connected to a server, Keyspace and Column Family names are not available.
 /tmp/logs/commitlog/CommitLog-1308876643288.log.header
 Keyspace Unknown:
   Cf id 0: 444
 /tmp/logs/commitlog/CommitLog-1308877711517.log.header
 Keyspace Unknown:
   Cf id 1: 68848763
 ...
 /tmp/logs/commitlog/CommitLog-1308944451460.log.header
 Keyspace Unknown:
   Cf id 1: 61074
 /tmp/logs/commitlog/CommitLog-1308945597471.log.header
 Keyspace Unknown:
   Cf id 1000: 43175492
   Cf id 1: 108483
 /tmp/logs/commitlog/CommitLog-1308946745380.log.header
 Keyspace Unknown:
   Cf id 1000: 239223
   Cf id 1: 172211
 /tmp/logs/commitlog/CommitLog-1308947888397.log.header
 Keyspace Unknown:
   Cf id 1001: 57595560
   Cf id 1: 816960
   Cf id 1000: 0
 {noformat}
 CF 0 is the Status / LocationInfo CF and 1 is the HintedHandof CF. I dont 
 have it now, but IIRC CFStats showed the LocationInfo CF with dirty ops. 
 I was able to repo a case where flushing the CF's did not mark the log 
 segments as obsolete (attached unit-test patch). Steps are:
 1. Write to cf1 and flush.
 2. Current log segment is marked as dirty at the CL position when the flush 
 started, CommitLog.discardCompletedSegmentsInternal()
 3. Do not write to cf1 again.
 4. Roll the log, my test does this manually. 
 5. Write to CF2 and flush.
 6. Only CF2 is flushed because it is the only dirty CF.

[jira] [Commented] (CASSANDRA-2979) Skip system table when repair

2011-08-01 Thread Norman Maurer (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073607#comment-13073607
]

Norman Maurer commented on CASSANDRA-2979:
--

I just had a quick look again and it seems like it also tries to repair the
system table in 0.8.x. Maybe I just miss something here so it will never try
todo so...

About the patch.. I was not sure if the right place is StorageService or
NodeProbe to handle this here. So I'm open for suggestions and to provide
another patch if necessary..

Skip system table when repair
-

Key: CASSANDRA-2979
URL: https://issues.apache.org/jira/browse/CASSANDRA-2979
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Norman Maurer
Assignee: Norman Maurer
Fix For: 0.8.3

Attachments: CASSANDRA-2979

From ml:
When I run nodetool repair on a node on my 3-node cluster, I see 3 messages
like the following:
INFO [manual-repair-6d9a617f-c496-4744-9002-a56909b83d5b] 2011-07-30
18:50:28,464 AntiEntropyService.java (line 636) No neighbors to repair with
for system on (0,56713727820156410577229101238628035242]:
manual-repair-6d9a617f-c496-4744-9002-a56909b83d5b completed.
Granted, there's no data on these machines yet. Is this normal?
And the answer:
You can discard those. This is saying it cannot repair the system keyspace
(table used internally). It turns those aren't replicated and thus don't need
to be repaired. We should probably update the code to skip trying to
repair the system table but in the meantime this is harmless.
--
Sylvain

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2967) Only bind JMX to the same IP address that is being used in Cassandra

2011-08-01 Thread Norman Maurer (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073609#comment-13073609
 ] 

Norman Maurer commented on CASSANDRA-2967:
--

We have something similar in JAMES. I will write up a patch for it.

 Only bind JMX to the same IP address that is being used in Cassandra
 

 Key: CASSANDRA-2967
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2967
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.8.2
Reporter: Joaquin Casares
Priority: Minor
  Labels: lhf

 The setup is 5 nodes in each data center are all running on one physical test 
 machine and even though the repair was run against the correct IP the wrong 
 JMX port was used. As a result, instead of repairing all 5 nodes I was 
 repairing the same node 5 times.
 It would be nice if Cassandra's JMX would bind to only the IP address on 
 which its thrift/RPC services are listening on instead of binding to all IP's 
 on the box.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2968) AssertionError during compaction of CF with counter columns


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2968:


Attachment: 2968.patch

This is actually a pretty stupid bug (not that there is smart bug): the old 
NodeId for the local node were read from the system table in reversed order 
while they shouldn't. The wrong path was then taken based on that mistake. No 
data was lost due to that (i.e, the total value of the counters is preserved), 
but non-sensical counter context were created (hence triggering the assertion).

Fixing the root cause is pretty straightforward. Fixing the nonsensical counter 
contexts is more subtle, but it is doable up to the fact that the local NodeId 
on the node(s) where the assertion is triggered will have to be renewed. 
Attaching a patch that does both (fixing root cause and repairing the bad 
data). Also add two unit tests, one for the root cause and one to check that 
the bad data repair code does what it is supposed to do.

After applying that patch (or upgrading on a release shipping it), you will 
(potentially) need to restart the node with the 
-Dcassandra.renew_counter_id=true (compaction will still fail if you don't but 
with a message saying that you should restart with the startup flag).

 AssertionError during compaction of CF with counter columns
 ---

 Key: CASSANDRA-2968
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2968
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.2
 Environment: CentOS release 5.6
Reporter: Taras Puchko
Assignee: Sylvain Lebresne
 Fix For: 0.8.3

 Attachments: 2968.patch, AffiliateActivity-g-147-Data.db, 
 AffiliateActivity-g-147-Index.db, AffiliateActivity-g-195-Data.db, 
 AffiliateActivity-g-195-Index.db


 Having upgraded from 0.8.0 to 0.8.2 we ran nodetool compact and got
 Error occured during compaction
 java.util.concurrent.ExecutionException: java.lang.AssertionError
 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
 at 
 org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762)
 at 
 org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
 at 
 com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
 at 
 com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
 at 
 com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
 at 
 com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
 at 
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
 at 
 javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
 at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
 at sun.rmi.transport.Transport$1.run(Transport.java:159)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
 at 
 sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
 at 
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
 at 
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
 at

[jira] [Commented] (CASSANDRA-2898) Escape characters in CQL

2011-08-01 Thread Blake Visin (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073617#comment-13073617
 ] 

Blake Visin commented on CASSANDRA-2898:


I did not explain exactly.  

Try 

cqlsh update foo set 'fmd:test' = 'foo' where key = 'test';
Unmatched named substitution: 'test' not given for update foo set 'fmd:test' = 
'foo' where key = 'test';

You can not escape a colon.


 Escape characters in CQL
 

 Key: CASSANDRA-2898
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2898
 Project: Cassandra
  Issue Type: Bug
  Components: Drivers
Reporter: Blake Visin
Assignee: Brandon Williams
  Labels: cql

 When trying to get all the columns named fmd: in cqlsh you can not escape : 
 or ;
 As per Jonathan Ellis:
 You can escape quotes but I don't think you can escape semicolons.
 Try:
 sqlsh select 'fmd:'..'fmd;' from feeds;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2972) nodetool netstats progress does not update on receiving side


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-2972:
--

Attachment: cassandra-0.8-2972.txt

 nodetool netstats progress does not update on receiving side
 

 Key: CASSANDRA-2972
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2972
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
Assignee: Yuki Morishita
Priority: Minor
 Attachments: cassandra-0.8-2972.txt


 when you add/remove node to cluster, nodetool netstats show correct results 
 only on sending side - on receiving side you can see only 0% progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2972) nodetool netstats progress does not update on receiving side


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073618#comment-13073618
 ] 

Yuki Morishita commented on CASSANDRA-2972:
---

Patch attached for 0.8 branch.
StreamInSession holds references to all pending files(files attribute) and 
current processing file(current attribute). The former include the latter but 
those are different objects after deserialized, and only the latter gets update 
on progress.
When returning incoming streaming status via JMX, only the former pending files 
described above are returned due to the nature of HashSet#add.

 nodetool netstats progress does not update on receiving side
 

 Key: CASSANDRA-2972
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2972
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
Assignee: Yuki Morishita
Priority: Minor
 Attachments: cassandra-0.8-2972.txt


 when you add/remove node to cluster, nodetool netstats show correct results 
 only on sending side - on receiving side you can see only 0% progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1152870 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/service/StorageService.java

Author: slebresne
Date: Mon Aug  1 17:33:35 2011
New Revision: 1152870

URL: http://svn.apache.org/viewvc?rev=1152870view=rev
Log:
Ignore system tables during repair
patch by norman; reviewed by slebresne for CASSANDRA-2979

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1152870r1=1152869r2=1152870view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Mon Aug  1 17:33:35 2011
@@ -21,6 +21,7 @@
with no post-flush activity permanently dirty) (CASSANDRA-2829)
  * fix throwing exception with batch mutation of counter super columns
(CASSANDRA-2949)
+ * ignore system tables during repair (CASSANDRA-2979)
 
 
 0.8.2

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java?rev=1152870r1=1152869r2=1152870view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
 Mon Aug  1 17:33:35 2011
@@ -1532,6 +1532,11 @@ public class StorageService implements I
  */
 public void forceTableRepair(final String tableName, final String... 
columnFamilies) throws IOException
 {
+if (Table.SYSTEM_TABLE.equals(tableName)) 
+{
+return;
+}
+
 ListAntiEntropyService.RepairSession sessions = new 
ArrayListAntiEntropyService.RepairSession();
 for (Range range : getLocalRanges(tableName))
 {

svn commit: r1152872 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/service/

Author: slebresne
Date: Mon Aug  1 17:34:45 2011
New Revision: 1152872

URL: http://svn.apache.org/viewvc?rev=1152872view=rev
Log:
merge from 0.8

Modified:
cassandra/trunk/   (props changed)
cassandra/trunk/CHANGES.txt
cassandra/trunk/contrib/   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
   (props changed)
cassandra/trunk/src/java/org/apache/cassandra/service/StorageService.java

Propchange: cassandra/trunk/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 17:34:45 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291
 /cassandra/branches/cassandra-0.7:1026516-1151306
 /cassandra/branches/cassandra-0.7.0:1053690-1055654
-/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1152110,1152265,1152793,1152795
+/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1152110,1152265,1152793,1152795,1152870
 /cassandra/branches/cassandra-0.8.0:1125021-1130369
 /cassandra/branches/cassandra-0.8.1:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689

Modified: cassandra/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1152872r1=1152871r2=1152872view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Mon Aug  1 17:34:45 2011
@@ -43,6 +43,7 @@
with no post-flush activity permanently dirty) (CASSANDRA-2829)
  * fix throwing exception with batch mutation of counter super columns
(CASSANDRA-2949)
+ * ignore system tables during repair (CASSANDRA-2979)
 
 
 0.8.2

Propchange: cassandra/trunk/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 17:34:45 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
 /cassandra/branches/cassandra-0.7/contrib:1026516-1151306
 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
-/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1152110,1152265,1152793,1152795
+/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1152110,1152265,1152793,1152795,1152870
 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369
 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 17:34:45 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1151306
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654
-/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1152110,1152265,1152793,1152795
+/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1152110,1152265,1152793,1152795,1152870
 
/cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369
 
/cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018
 
/cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Aug  1 17:34:45 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1151306

[jira] [Commented] (CASSANDRA-2960) replication_factor 1 always causes cassandra to return null


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073622#comment-13073622
 ] 

Sylvain Lebresne commented on CASSANDRA-2960:
-

+1

 replication_factor  1 always causes cassandra to return null
 -

 Key: CASSANDRA-2960
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2960
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: Ubuntu 11.04
Reporter: Steve Corona
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.8.3

 Attachments: 2960.txt


 On a brand new cluster:
   
 [default@SimpleTest] create keyspace SimpleTest2 with strategy_options = 
 [{replication_factor:3}];  
 16babc60-b886-11e0--c9ff69cb2dfb
 Waiting for schema agreement...
 ... schemas agree across the cluster
 [default@SimpleTest] use SimpleTest2;
 Authenticated to keyspace: SimpleTest2
 [default@SimpleTest2] create column family CFTest with comparator=UTF8Type 
 and default_validation_class=UTF8Type;
 1f108660-b886-11e0--c9ff69cb2dfb
 Waiting for schema agreement...
 ... schemas agree across the cluster
 [default@SimpleTest2] set CFTest['1']['text'] = 'test';
 null
 [default@SimpleTest2] get CFTest['1'];
 null
 [default@SimpleTest2] list CFTest;
 Using default limit of 100
 null
 [default@SimpleTest2] describe cluster;
 Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions: 
   1f108660-b886-11e0--c9ff69cb2dfb: [10.60.98.20, 10.60.98.24, 
 10.60.98.26]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown when durable writes are disabled

2011-08-01 Thread David Phillips (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073624#comment-13073624
 ] 

David Phillips commented on CASSANDRA-2958:
---

It looks like waitOnFutures() can return a null pointer exception if 
forceFlush() returns null?

 Flush memtables on shutdown when durable writes are disabled
 

 Key: CASSANDRA-2958
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: David Phillips
Assignee: Jonathan Ellis
 Fix For: 0.8.3

 Attachments: 2958.txt


 Memtables need to be flushed on shutdown when durable_writes is set to false, 
 otherwise data loss occurs as the data is not available to be replayed from 
 the commit log.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (CASSANDRA-2958) Flush memtables on shutdown when durable writes are disabled

2011-08-01 Thread David Phillips (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Phillips reopened CASSANDRA-2958:
---


 Flush memtables on shutdown when durable writes are disabled
 

 Key: CASSANDRA-2958
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: David Phillips
Assignee: Jonathan Ellis
 Fix For: 0.8.3

 Attachments: 2958.txt


 Memtables need to be flushed on shutdown when durable_writes is set to false, 
 otherwise data loss occurs as the data is not available to be replayed from 
 the commit log.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2972) nodetool netstats progress does not update on receiving side


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-2972:
--

Attachment: cassandra-0.8-2972-v2.txt

 nodetool netstats progress does not update on receiving side
 

 Key: CASSANDRA-2972
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2972
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
Assignee: Yuki Morishita
Priority: Minor
 Attachments: cassandra-0.8-2972-v2.txt, cassandra-0.8-2972.txt


 when you add/remove node to cluster, nodetool netstats show correct results 
 only on sending side - on receiving side you can see only 0% progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2972) nodetool netstats progress does not update on receiving side


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073626#comment-13073626
 ] 

Yuki Morishita commented on CASSANDRA-2972:
---

Slightly modified version.

 nodetool netstats progress does not update on receiving side
 

 Key: CASSANDRA-2972
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2972
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
Assignee: Yuki Morishita
Priority: Minor
 Attachments: cassandra-0.8-2972-v2.txt, cassandra-0.8-2972.txt


 when you add/remove node to cluster, nodetool netstats show correct results 
 only on sending side - on receiving side you can see only 0% progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1152876 - /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java

2011-08-01 Thread brandonwilliams

Author: brandonwilliams
Date: Mon Aug  1 17:54:50 2011
New Revision: 1152876

URL: http://svn.apache.org/viewvc?rev=1152876view=rev
Log:
Don't sample the system table keys when choosing a bootstrap token.
Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-2825

Modified:

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java?rev=1152876r1=1152875r2=1152876view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
 Mon Aug  1 17:54:50 2011
@@ -1759,6 +1759,8 @@ public class StorageService implements I
 ListDecoratedKey keys = new ArrayListDecoratedKey();
 for (ColumnFamilyStore cfs : ColumnFamilyStore.all())
 {
+if (cfs.table.name.equals(Table.SYSTEM_TABLE))
+continue;
 for (DecoratedKey key : cfs.allKeySamples())
 {
 if (range.contains(key.token))
@@ -1767,9 +1769,19 @@ public class StorageService implements I
 }
 FBUtilities.sortSampledKeys(keys, range);
 
-Token token = keys.size()  3
-? partitioner.midpoint(range.left, range.right)
-: keys.get(keys.size() / 2).token;
+Token token;
+if (keys.size()  3)
+{
+token = partitioner.midpoint(range.left, range.right);
+logger_.debug(Used midpoint to assign token  + token);
+}
+else
+{
+token = keys.get(keys.size() / 2).token;
+logger_.debug(Used key sample of size  + keys.size() +  to 
assign token  + token);
+}
+if (tokenMetadata_.getEndpoint(token) != null  
tokenMetadata_.isMember(tokenMetadata_.getEndpoint(token)))
+throw new RuntimeException(Chose token  + token +  which is 
already in use by  + tokenMetadata_.getEndpoint(token) +  -- specify one 
manually with initial_token);
 // Hack to prevent giving nodes tokens with DELIMITER_STR in them 
(which is fine in a row key/token)
 if (token instanceof StringToken)
 {

[jira] [Commented] (CASSANDRA-2506) Push read repair setting down to the DC-level

2011-08-01 Thread Ryan King (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073634#comment-13073634
 ] 

Ryan King commented on CASSANDRA-2506:
--

It would also be nice if you could specify a different repair rate for intra-DC 
and inter-DC repairs.

 Push read repair setting down to the DC-level
 -

 Key: CASSANDRA-2506
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2506
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Brandon Williams
Assignee: Patricio Echague
 Fix For: 1.0


 Currently, read repair is a global setting.  However, when you have two DCs 
 and use one for analytics, it would be nice to turn it off only for that DC 
 so the live DC serving the application can still benefit from it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2498) Improve read performance in update-intensive workload

2011-08-01 Thread Ryan King (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073633#comment-13073633
 ] 

Ryan King commented on CASSANDRA-2498:
--

Not sure what we can do about that unless we make counters idempotent, which 
may not feasible.

 Improve read performance in update-intensive workload
 -

 Key: CASSANDRA-2498
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2498
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Sylvain Lebresne
Priority: Minor
  Labels: ponies
 Fix For: 1.0

 Attachments: 2498-v2.txt, supersede-name-filter-collations.patch


 Read performance in an update-heavy environment relies heavily on compaction 
 to maintain good throughput. (This is not the case for workloads where rows 
 are only inserted once, because the bloom filter keeps us from having to 
 check sstables unnecessarily.)
 Very early versions of Cassandra attempted to mitigate this by checking 
 sstables in descending generation order (mostly equivalent to descending 
 mtime): once all the requested columns were found, it would not check any 
 older sstables.
 This was incorrect, because data timestamp will not correspond to sstable 
 timestamp, both because compaction has the side effect of refreshing data 
 to a newer sstable, and because hintead handoff may send us data older than 
 what we already have.
 Instead, we could create a per-sstable piece of metadata containing the most 
 recent (client-specified) timestamp for any column in the sstable.  We could 
 then sort sstables by this timestamp instead, and perform a similar 
 optimization (if the remaining sstable client-timestamps are older than the 
 oldest column found in the desired result set so far, we don't need to look 
 further). Since under almost every workload, client timestamps of data in a 
 given sstable will tend to be similar, we expect this to cut the number of 
 sstables down proportionally to how frequently each column in the row is 
 updated. (If each column is updated with each write, we only have to check a 
 single sstable.)
 This may also be useful information when deciding which SSTables to compact.
 (Note that this optimization is only appropriate for named-column queries, 
 not slice queries, since we don't know what non-overlapping columns may exist 
 in older sstables.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2868) Native Memory Leak

[
https://issues.apache.org/jira/browse/CASSANDRA-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073636#comment-13073636
]

Sylvain Lebresne commented on CASSANDRA-2868:
-

Comments on v2:
* Couldn't we estimate the reclaimed size by recording the last memory used
(that would need to be the first thing we do in logGCResults so that we record
it each time) ?
* Wouldn't it be worth indicating that how many collection have been done since
last log message if it's 1, since it can (be 1).
* Nit: especially if we decide to keep the last memory used, it may be more
efficient (in cleaner imho) to have just one HashMap of string - GCInfo where
GCInfo would be a small struct with times, counts and usedMemory. Not that it
is very performance sensitive...

Native Memory Leak
--

Key: CASSANDRA-2868
URL: https://issues.apache.org/jira/browse/CASSANDRA-2868
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Daniel Doubleday
Assignee: Brandon Williams
Priority: Minor
Fix For: 0.8.3

Attachments: 2868-v1.txt, 2868-v2.txt, 48hour_RES.png,
low-load-36-hours-initial-results.png

We have memory issues with long running servers. These have been confirmed by
several users in the user list. That's why I report.
The memory consumption of the cassandra java process increases steadily until
it's killed by the os because of oom (with no swap)
Our server is started with -Xmx3000M and running for around 23 days.
pmap -x shows
Total SST: 1961616 (mem mapped data and index files)
Anon RSS: 6499640
Total RSS: 8478376
This shows that 3G are 'overallocated'.
We will use BRAF on one of our less important nodes to check wether it is
related to mmap and report back.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Attachment: CASSANDRA-2034-trunk.patch

Attaching a preliminary patch.

I still need to handle hints on failure.

Please have a look at how the timeouts are handled and if that part needs to be 
improved. 

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2825) Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no token explicitly assigned in config.


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073642#comment-13073642
 ] 

Hudson commented on CASSANDRA-2825:
---

Integrated in Cassandra-0.8 #249 (See 
[https://builds.apache.org/job/Cassandra-0.8/249/])
Don't sample the system table keys when choosing a bootstrap token.
Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-2825

brandonwilliams : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1152876
Files : 
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java


 Auto bootstrapping the 4th node in a 4 node cluster doesn't work, when no 
 token explicitly assigned in config.
 --

 Key: CASSANDRA-2825
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2825
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0, 0.8.1
Reporter: Michael Allen
Assignee: Brandon Williams
 Fix For: 0.8.3

 Attachments: 2825-v2.txt, 2825-v3.txt, 2825.txt


 This was done in sequence.  A, B, C, and D.  Node A with token 0 explicitly 
 set in config.  The rest with auto_bootstrap: true and no token explicitly 
 assigned.  B and C work as expected. D ends up stealing C's token.  
 from system.log on C:
 INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 638) Node 
 /10.171.47.226 is now part of the cluster
 INFO [GossipStage:1] 2011-06-24 16:40:41,947 Gossiper.java (line 606) 
 InetAddress /10.171.47.226 is now UP
 INFO [GossipStage:1] 2011-06-24 16:42:09,432 StorageService.java (line 769) 
 Nodes /10.171.47.226 and /10.171.55.77 have the same token 
 61078635599166706937511052402724559481.  /10.171.47.226 is the new owner
 WARN [GossipStage:1] 2011-06-24 16:42:09,432 TokenMetadata.java (line 120) 
 Token 61078635599166706937511052402724559481 changing ownership from 
 /10.171.55.77 to /10.171.47.226

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2979) Skip system table when repair


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073641#comment-13073641
 ] 

Hudson commented on CASSANDRA-2979:
---

Integrated in Cassandra-0.8 #249 (See 
[https://builds.apache.org/job/Cassandra-0.8/249/])
Ignore system tables during repair
patch by norman; reviewed by slebresne for CASSANDRA-2979

slebresne : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1152870
Files : 
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java


 Skip system table when repair
 -

 Key: CASSANDRA-2979
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2979
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Norman Maurer
Assignee: Norman Maurer
 Fix For: 0.8.3

 Attachments: CASSANDRA-2979


 From ml:
 When I run nodetool repair on a node on my 3-node cluster, I see 3 messages 
 like the following:
  INFO [manual-repair-6d9a617f-c496-4744-9002-a56909b83d5b] 2011-07-30 
 18:50:28,464 AntiEntropyService.java (line 636) No neighbors to repair with 
 for system on (0,56713727820156410577229101238628035242]: 
 manual-repair-6d9a617f-c496-4744-9002-a56909b83d5b completed.
 Granted, there's no data on these machines yet.  Is this normal?
 And the answer:
 You can discard those. This is saying it cannot repair the system keyspace
 (table used internally). It turns those aren't replicated and thus don't need
 to be repaired. We should probably update the code to skip trying to
 repair the system table but in the meantime this is harmless.
 --
 Sylvain

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1152890 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java

2011-08-01 Thread jbellis

Author: jbellis
Date: Mon Aug  1 18:42:09 2011
New Revision: 1152890

URL: http://svn.apache.org/viewvc?rev=1152890view=rev
Log:
throw exception when NTS is given replication_factoras an option
patch by jbellis; reviewed by slebresne for CASSANDRA-2960

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1152890r1=1152889r2=1152890view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Mon Aug  1 18:42:09 2011
@@ -17,11 +17,13 @@
(CASSANDRA-2958)
  * improved POSIX compatibility of start scripts (CASsANDRA-2965)
  * add counter support to Hadoop InputFormat (CASSANDRA-2981)
- * fix bug where dirty commit logs were removed (and avoid keeping segment
+ * fix bug where dirty commit logs were removed (and avoid keeping segments
with no post-flush activity permanently dirty) (CASSANDRA-2829)
  * fix throwing exception with batch mutation of counter super columns
(CASSANDRA-2949)
  * ignore system tables during repair (CASSANDRA-2979)
+ * throw exception when NTS is given replication_factor as an option
+   (CASSANDRA-2960)
 
 
 0.8.2

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java?rev=1152890r1=1152889r2=1152890view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
 Mon Aug  1 18:42:09 2011
@@ -67,7 +67,7 @@ public class NetworkTopologyStrategy ext
 {
 String dc = entry.getKey();
 if (dc.equalsIgnoreCase(replication_factor))
-continue; // TODO remove this for 1.0
+throw new ConfigurationException(replication_factor is an 
option for SimpleStrategy, not NetworkTopologyStrategy);
 Integer replicas = Integer.valueOf(entry.getValue());
 newDatacenters.put(dc, replicas);
 }

[jira] [Commented] (CASSANDRA-1379) Uncached row reads may block cached reads

2011-08-01 Thread Ryan King (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073653#comment-13073653
 ] 

Ryan King commented on CASSANDRA-1379:
--

We have use cases along these lines too. We've had to resort to bumping the 
read threads up much higher (128 or 256) for highly cached workloads.

 Uncached row reads may block cached reads
 -

 Key: CASSANDRA-1379
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1379
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: David King
Assignee: Javier Canillas
Priority: Minor
 Attachments: CASSANDRA-1379.patch


 The cap on the number of concurrent reads appears to cap the *total* number 
 of concurrent reads instead of just capping the reads that are bound for 
 disk. That is, given N concurrent readers if all of them are busy waiting on 
 disk, even reads that can be served from the row cache will block waiting for 
 them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1152891 - /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java

2011-08-01 Thread jbellis

Author: jbellis
Date: Mon Aug  1 18:44:41 2011
New Revision: 1152891

URL: http://svn.apache.org/viewvc?rev=1152891view=rev
Log:
avoid NPE when flushing in shutdown hook
patch by jbellis for CASSANDRA-2958

Modified:

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java?rev=1152891r1=1152890r2=1152891view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
 Mon Aug  1 18:44:41 2011
@@ -417,7 +417,11 @@ public class StorageService implements I
 if (!ksm.isDurableWrites())
 {
 for (ColumnFamilyStore cfs : 
table.getColumnFamilyStores())
-flushes.add(cfs.forceFlush());
+{
+Future? future = cfs.forceFlush();
+if (future != null)
+flushes.add(future);
+}
 }
 }
 FBUtilities.waitOnFutures(flushes);

[jira] [Resolved] (CASSANDRA-2958) Flush memtables on shutdown when durable writes are disabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis resolved CASSANDRA-2958.
---

Resolution: Fixed

thanks -- fixed in r1152891

 Flush memtables on shutdown when durable writes are disabled
 

 Key: CASSANDRA-2958
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: David Phillips
Assignee: Jonathan Ellis
 Fix For: 0.8.3

 Attachments: 2958.txt


 Memtables need to be flushed on shutdown when durable_writes is set to false, 
 otherwise data loss occurs as the data is not available to be replayed from 
 the commit log.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2868) Native Memory Leak

[
https://issues.apache.org/jira/browse/CASSANDRA-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073661#comment-13073661
]

Jonathan Ellis commented on CASSANDRA-2868:
---

bq. Couldn't we estimate the reclaimed size

Well, not really, what we'd have is difference in size between last time it
was called, and now which isn't all that close to amount reclaimed by a
specific GC.

bq. Wouldn't it be worth indicating that how many collection have been done
since last log message

IMO the duration-based thresholds are hard to reason about here, where we're
dealing w/ summaries and not individual GC results. I think I'd rather have
something like the dropped messages logger, where every N seconds we log the
summary we get from the mbean.

The flushLargestMemtables/reduceCacheSizes stuff should probably be removed. :(

Native Memory Leak
--

Attachments: 2868-v1.txt, 2868-v2.txt, 48hour_RES.png,
low-load-36-hours-initial-results.png

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2982) Refactor secondary index api

2011-08-01 Thread T Jake Luciani (JIRA)

Refactor secondary index api


 Key: CASSANDRA-2982
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2982
 Project: Cassandra
  Issue Type: Sub-task
  Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
 Fix For: 1.0


Secondary indexes currently make some bad assumptions about the underlying 
indexes.

1. That they are always stored in other column families.
2. That there is a unique index per column

In the case of CASSANDRA-2915 neither of these are true.  The new api should 
abstract the search concepts and allow any search api to plug in.

Once the code is refactored and basically pluggable we can remove the IndexType 
enum and use class names similar to how we handle partitioners and comparators.

Basic api is to add a SecondaryIndexManager that handles different index types 
per CF and a SecondaryIndex base class that handles a particular type 
implementation.

This requires major changes to ColumnFamilyStore and Table.IndexBuilder

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown when durable writes are disabled


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073671#comment-13073671
 ] 

Hudson commented on CASSANDRA-2958:
---

Integrated in Cassandra-0.8 #250 (See 
[https://builds.apache.org/job/Cassandra-0.8/250/])
avoid NPE when flushing in shutdown hook
patch by jbellis for CASSANDRA-2958

jbellis : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1152891
Files : 
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java


 Flush memtables on shutdown when durable writes are disabled
 

 Key: CASSANDRA-2958
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: David Phillips
Assignee: Jonathan Ellis
 Fix For: 0.8.3

 Attachments: 2958.txt


 Memtables need to be flushed on shutdown when durable_writes is set to false, 
 otherwise data loss occurs as the data is not available to be replayed from 
 the commit log.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2960) replication_factor 1 always causes cassandra to return null


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073672#comment-13073672
 ] 

Hudson commented on CASSANDRA-2960:
---

Integrated in Cassandra-0.8 #250 (See 
[https://builds.apache.org/job/Cassandra-0.8/250/])
throw exception when NTS is given replication_factoras an option
patch by jbellis; reviewed by slebresne for CASSANDRA-2960

jbellis : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1152890
Files : 
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java


 replication_factor  1 always causes cassandra to return null
 -

 Key: CASSANDRA-2960
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2960
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: Ubuntu 11.04
Reporter: Steve Corona
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.8.3

 Attachments: 2960.txt


 On a brand new cluster:
   
 [default@SimpleTest] create keyspace SimpleTest2 with strategy_options = 
 [{replication_factor:3}];  
 16babc60-b886-11e0--c9ff69cb2dfb
 Waiting for schema agreement...
 ... schemas agree across the cluster
 [default@SimpleTest] use SimpleTest2;
 Authenticated to keyspace: SimpleTest2
 [default@SimpleTest2] create column family CFTest with comparator=UTF8Type 
 and default_validation_class=UTF8Type;
 1f108660-b886-11e0--c9ff69cb2dfb
 Waiting for schema agreement...
 ... schemas agree across the cluster
 [default@SimpleTest2] set CFTest['1']['text'] = 'test';
 null
 [default@SimpleTest2] get CFTest['1'];
 null
 [default@SimpleTest2] list CFTest;
 Using default limit of 100
 null
 [default@SimpleTest2] describe cluster;
 Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions: 
   1f108660-b886-11e0--c9ff69cb2dfb: [10.60.98.20, 10.60.98.24, 
 10.60.98.26]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Attachment: CASSANDRA-2034-trunk-v2.patch

Replaces first version

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk-v2.patch, CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF

[
https://issues.apache.org/jira/browse/CASSANDRA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-2901:
--

Attachment: 2901-v3.txt

bq. PCI.Reducer.getCompactedRow unwraps NotifyingSSTableIterator

fixed

bq. it doesn't seem like we ever call close() on the SSTableIdentityIterator

fixed

bq. we need to reuse the trick of DebuggableThreadPoolExecutor

added ACR.close to fix

bq. we already queue up rows in the MergeTask executor, so it feels like it
would be simple to use direct handoff here

Simpler, but then any deserializer that can't deserialize the next row by the
time the next merge slot is open, will stall the merge. My thinking was that
allowing per-deserializer buffers will keep the pipeline full better.

bq. the memory blow up is (potentially) much more than the 2x (compared to
mono-threaded) in the description of this ticket

It's not so bad, because we can assume N = 2 sstables, and we restrict each
deserializer to 1/N of in-memory limit. So I think we come close to 2x
overall. (And if we don't, I'd rather adjust our estimate, than make it less
performant/more complex. :)

bq. we should probably remove the occurrence in PreCompactedRow as it's still
multi-threaded while in the MergeTask

refactored PCR constructors to do this.

bq. we create one ComparatorIColumnIterator each time instead of having a
private static final one

added RowContainer.comparator

bq. there is a race when updating the bytesRead

fixed, and changed bytesRead to volatile

Allow taking advantage of multiple cores while compacting a single CF
-

Attachments: 2901-v2.txt, 2901-v3.txt, 2901.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Comment: was deleted

(was: Replaces first version)

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Attachment: (was: CASSANDRA-2034-trunk.patch)

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Attachment: (was: CASSANDRA-2034-trunk-v2.patch)

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Attachment: CASSANDRA-2034-trunk.patch

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2819) Split rpc timeout for read and write ops


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073781#comment-13073781
 ] 

Melvin Wang commented on CASSANDRA-2819:


In MS.timeoutReporter, we blindly add default rpc timeout as the latency. Since 
different messages may have different timeout now, in ExpiringMap.CacheMonitor, 
probably we need to pass the individual expiration time when we call apply().

 Split rpc timeout for read and write ops
 

 Key: CASSANDRA-2819
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2819
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Stu Hood
Assignee: Melvin Wang
 Fix For: 1.0

 Attachments: 2819-v4.txt, rpc-jira.patch


 Given the vastly different latency characteristics of reads and writes, it 
 makes sense for them to have independent rpc timeouts internally.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2898) Escape characters in CQL


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-2898:


Attachment: 2898.txt

Patch to allow escaping colons with a backslash.

 Escape characters in CQL
 

 Key: CASSANDRA-2898
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2898
 Project: Cassandra
  Issue Type: Bug
  Components: Drivers
Reporter: Blake Visin
Assignee: Brandon Williams
  Labels: cql
 Attachments: 2898.txt


 When trying to get all the columns named fmd: in cqlsh you can not escape : 
 or ;
 As per Jonathan Ellis:
 You can escape quotes but I don't think you can escape semicolons.
 Try:
 sqlsh select 'fmd:'..'fmd;' from feeds;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2898) Escape characters in CQL


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-2898:


Attachment: (was: 2898.txt)

 Escape characters in CQL
 

 Key: CASSANDRA-2898
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2898
 Project: Cassandra
  Issue Type: Bug
  Components: Drivers
Reporter: Blake Visin
Assignee: Brandon Williams
  Labels: cql
 Attachments: 2898.txt


 When trying to get all the columns named fmd: in cqlsh you can not escape : 
 or ;
 As per Jonathan Ellis:
 You can escape quotes but I don't think you can escape semicolons.
 Try:
 sqlsh select 'fmd:'..'fmd;' from feeds;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled

[
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073791#comment-13073791
]

Jonathan Ellis commented on CASSANDRA-2034:
---

Looks reasonable. I'd move the wait method into FBUtilities as an overload of
waitOnFutures.

Does the timeout on get start when the future is created, or when get is
called? I think it is the latter but I am not sure. If so, we should track
the total time waited and reduce after each get() so we do not allow total of
up to timeout * hints.

Make Read Repair unnecessary when Hinted Handoff is enabled
---

Key: CASSANDRA-2034
URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
Fix For: 1.0

Attachments: CASSANDRA-2034-trunk.patch

Original Estimate: 8h
Remaining Estimate: 8h

Currently, HH is purely an optimization -- if a machine goes down, enabling
HH means RR/AES will have less work to do, but you can't disable RR entirely
in most situations since HH doesn't kick in until the FailureDetector does.
Let's add a scheduled task to the mutate path, such that we return to the
client normally after ConsistencyLevel is achieved, but after RpcTimeout we
check the responseHandler write acks and write local hints for any missing
targets.
This would making disabling RR when HH is enabled a much more reasonable
option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2819) Split rpc timeout for read and write ops


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073795#comment-13073795
 ] 

Melvin Wang commented on CASSANDRA-2819:


looks like readcallback, repaircallback, abstractWriteresponseHandler all 
assume the timeout value from getRpcTimeout(); however with this patch, 
ExpiringMap will timeout different messages with different values, so we may 
have situation where expiringMap already expires the message while the get() 
still blocks if we have different read_rpc_timeout/write_rpc_timeout with the 
'default' rpc timeout. I'm wondering if we could modify the IAsyncCallback so 
that we expiringMap expires a message, it will signal the callback and mark it 
as a failure, and hence in get() we don't need to have the logic to track the 
timeout.

 Split rpc timeout for read and write ops
 

 Key: CASSANDRA-2819
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2819
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Stu Hood
Assignee: Melvin Wang
 Fix For: 1.0

 Attachments: 2819-v4.txt, rpc-jira.patch


 Given the vastly different latency characteristics of reads and writes, it 
 makes sense for them to have independent rpc timeouts internally.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled

[
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073807#comment-13073807
]

Patricio Echague commented on CASSANDRA-2034:
-

{quote} Does the timeout on get start when the future is created, or when get
is called? I think it is the latter but I am not sure. If so, we should track
the total time waited and reduce after each get() so we do not allow total of
up to timeout * hints. {quote}
Yeah, I need to add a creation time and so something similar to what
IAsynResult does.

I noticed I missed to skip the hints creation when HH is disabled.
Some thoughts on this I would like some feedback:

Note: remember that hints are written locally on the coordinator node now.

| Hinted Handoff | Consist. Level |
| on | =1 | -- wait for hints. We DO NOT notify the
handler with handler.response() for hints;
| on | ANY | -- wait for hints. Responses count towards
consistency.
| off| =1 | -- DO NOT fire hints. And DO NOT wait for
them to complete.
| off| ANY | -- Fire hints but don't wait for them.
They count towards consistency.

Make Read Repair unnecessary when Hinted Handoff is enabled
---

Attachments: CASSANDRA-2034-trunk.patch

Original Estimate: 8h
Remaining Estimate: 8h

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073812#comment-13073812
 ] 

Jonathan Ellis commented on CASSANDRA-2034:
---

Looks good to me for the first 3.

I think ANY should be equal to ONE for hints=off.  I.e., when it's off we 
*never* create hints.

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2941) Expose number of rpc timeouts for individual hosts metric via jmx


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melvin Wang updated CASSANDRA-2941:
---

Attachment: (was: 
twttr-cassandra-0.8-counts-resync-timeouts-metric.patch)

 Expose number of rpc timeouts for individual hosts metric via jmx 
 --

 Key: CASSANDRA-2941
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2941
 Project: Cassandra
  Issue Type: Improvement
Reporter: Melvin Wang
Assignee: Melvin Wang
Priority: Minor

 We have a total number timeouts for each node. It's better for monitoring to 
 break down this total number into number of timeouts per host that this node 
 tried to connect to.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2975) Upgrade MurmurHash to version 3

2011-08-01 Thread Brian Lindauer (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073832#comment-13073832
 ] 

Brian Lindauer commented on CASSANDRA-2975:
---

You weren't kidding about compatibility with old data files not being simple. 
It actually turned out to be fairly major surgery. The original changes just to 
support Mumur3 are here:

https://github.com/lindauer/cassandra/commit/cea6068a4a3e5d7d9509335394f9ef3350d37e93

The additional proposed changes to support backward compatibility are at:

https://github.com/lindauer/cassandra/commit/9d7479675752a07732f434b307be6642d8b3e85f

I can't say I'm completely satisfied with these changes. It feels like we 
should unify with LegacyBloomFilter now that there are 3 versions. It also 
feels like all of the places where a serializer is selected based on a 
Descriptor version/flag could be moved under one roof, where callers just pass 
the Descriptor and it returns the correct serializer instance. But, not being 
too familiar with Cassandra, I was trying to be minimally invasive for fear of 
breaking something.

All of the tests pass, but I haven't added any tests, such as making sure that 
old files can still be read in. Like I said, I'm not very familiar with 
Cassandra, so you should review these changes carefully. (I'm sure you would 
anyway.)


 Upgrade MurmurHash to version 3
 ---

 Key: CASSANDRA-2975
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2975
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.8.3
Reporter: Brian Lindauer
Priority: Trivial
  Labels: lhf

 MurmurHash version 3 was finalized on June 3. It provides an enormous speedup 
 and increased robustness over version 2, which is implemented in Cassandra. 
 Information here:
 http://code.google.com/p/smhasher/
 The reference implementation is here:
 http://code.google.com/p/smhasher/source/browse/trunk/MurmurHash3.cpp?spec=svn136r=136
 I have already done the work to port the (public domain) reference 
 implementation to Java in the MurmurHash class and updated the BloomFilter 
 class to use the new implementation:
 https://github.com/lindauer/cassandra/commit/cea6068a4a3e5d7d9509335394f9ef3350d37e93
 Apart from the faster hash time, the new version only requires one call to 
 hash() rather than 2, since it returns 128 bits of hash instead of 64.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2941) Expose number of rpc timeouts for individual hosts metric via jmx


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melvin Wang updated CASSANDRA-2941:
---

Attachment: c2941.patch

Sorry for the confusion. This patch worked against the current trunk.

 Expose number of rpc timeouts for individual hosts metric via jmx 
 --

 Key: CASSANDRA-2941
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2941
 Project: Cassandra
  Issue Type: Improvement
Reporter: Melvin Wang
Assignee: Melvin Wang
Priority: Minor
 Attachments: c2941.patch


 We have a total number timeouts for each node. It's better for monitoring to 
 break down this total number into number of timeouts per host that this node 
 tried to connect to.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2975) Upgrade MurmurHash to version 3


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2975:
--

 Reviewer: xedin
Affects Version/s: (was: 0.8.3)
Fix Version/s: 1.0

 Upgrade MurmurHash to version 3
 ---

 Key: CASSANDRA-2975
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2975
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Brian Lindauer
Priority: Trivial
  Labels: lhf
 Fix For: 1.0


 MurmurHash version 3 was finalized on June 3. It provides an enormous speedup 
 and increased robustness over version 2, which is implemented in Cassandra. 
 Information here:
 http://code.google.com/p/smhasher/
 The reference implementation is here:
 http://code.google.com/p/smhasher/source/browse/trunk/MurmurHash3.cpp?spec=svn136r=136
 I have already done the work to port the (public domain) reference 
 implementation to Java in the MurmurHash class and updated the BloomFilter 
 class to use the new implementation:
 https://github.com/lindauer/cassandra/commit/cea6068a4a3e5d7d9509335394f9ef3350d37e93
 Apart from the faster hash time, the new version only requires one call to 
 hash() rather than 2, since it returns 128 bits of hash instead of 64.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (CASSANDRA-2975) Upgrade MurmurHash to version 3


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis reassigned CASSANDRA-2975:
-

Assignee: Brian Lindauer

 Upgrade MurmurHash to version 3
 ---

 Key: CASSANDRA-2975
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2975
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Brian Lindauer
Assignee: Brian Lindauer
Priority: Trivial
  Labels: lhf
 Fix For: 1.0


 MurmurHash version 3 was finalized on June 3. It provides an enormous speedup 
 and increased robustness over version 2, which is implemented in Cassandra. 
 Information here:
 http://code.google.com/p/smhasher/
 The reference implementation is here:
 http://code.google.com/p/smhasher/source/browse/trunk/MurmurHash3.cpp?spec=svn136r=136
 I have already done the work to port the (public domain) reference 
 implementation to Java in the MurmurHash class and updated the BloomFilter 
 class to use the new implementation:
 https://github.com/lindauer/cassandra/commit/cea6068a4a3e5d7d9509335394f9ef3350d37e93
 Apart from the faster hash time, the new version only requires one call to 
 hash() rather than 2, since it returns 128 bits of hash instead of 64.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2643) read repair/reconciliation breaks slice based iteration at QUORUM


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073980#comment-13073980
 ] 

Jonathan Ellis commented on CASSANDRA-2643:
---

Looks good on the whole.  One point to clear up:

{code}
if ((maxLiveColumns = sliceCommand.count)  (liveColumnsInRow  
sliceCommand.count))
{code}

maxLiveColumns is the max from a single response, so how can it be greater than 
sliceCommand.count?  Would this be a valid reformulation?

{code}
assert maxLiveColumns = sliceCommand.count;
if ((maxLiveColumns == sliceCommand.count)  (liveColumnsInRow  
sliceCommand.count))
{code}

Minor things I'd like to clean up:

- is maxLiveColumns valid on any AbstractRR subclass other than RRR? If not I'd 
rather move it in there and throw an UnsupportedOperation in ARR.
- Would prefer initializing commandsToRetry to Collections.emptyList, to avoid 
allocating that list in the common case that no retries are needed.  (Then 
clear of course needs to become allocate.)


 read repair/reconciliation breaks slice based iteration at QUORUM
 -

 Key: CASSANDRA-2643
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2643
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.5
Reporter: Peter Schuller
Assignee: Brandon Williams
Priority: Critical
 Fix For: 1.0

 Attachments: CASSANDRA-2643-poc.patch, CASSANDRA-2643-v2.patch, 
 reliable_short_read_0.8.sh, short_read.sh, short_read_0.8.sh, slicetest.py


 In short, I believe iterating over columns is impossible to do reliably with 
 QUORUM due to the way reconciliation works.
 The problem is that the SliceQueryFilter is executing locally when reading on 
 a node, but no attempts seem to be made to consider limits when doing 
 reconciliation and/or read-repair (RowRepairResolver.resolveSuperset() and 
 ColumnFamily.resolve()).
 If a node slices and comes up with 100 columns, and another node slices and 
 comes up with 100 columns, some of which are unique to each side, 
 reconciliation results in  100 columns in the result set. In this case the 
 effect is limited to client gets more than asked for, but the columns still 
 accurately represent the range. This is easily triggered by my test-case.
 In addition to the client receiving too many columns, I believe some of 
 them will not be satisfying the QUORUM consistency level for the same reasons 
 as with deletions (see discussion below).
 Now, there *should* be a problem for tombstones as well, but it's more 
 subtle. Suppose A has:
   1
   2
   3
   4
   5
   6
 and B has:
   1
   del 2
   del 3
   del 4
   5
   6 
 If you now slice 1-6 with count=3 the tombstones from B will reconcile with 
 those from A - fine. So you end up getting 1,5,6 back. This made it a bit 
 difficult to trigger in a test case until I realized what was going on. At 
 first I was hoping to see a short iteration result, which would mean that 
 the process of iterating until you get a short result will cause spurious 
 end of columns and thus make it impossible to iterate correctly.
 So; due to 5-6 existing (and if they didn't, you legitimately reached 
 end-of-columns) we do indeed get a result of size 3 which contains 1,5 and 6. 
 However, only node B would have contributed columns 5 and 6; so there is 
 actually no QUORUM consistency on the co-ordinating node with respect to 
 these columns. If node A and C also had 5 and 6, they would not have been 
 considered.
 Am I wrong?
 In any case; using script I'm about to attach, you can trigger the 
 over-delivery case very easily:
 (0) disable hinted hand-off to avoid that interacting with the test
 (1) start three nodes
 (2) create ks 'test' with rf=3 and cf 'slicetest'
 (3) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (4) stop node A
 (5) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (6) start node A, wait for B and C to consider it up
 (7) ./slicetest.py hostname_of_node_A slice # make A co-ordinator though it 
 doesn't necessarily matter
 You can also pass 'delete' (random deletion of 50% of contents) or 
 'deleterange' (delete all in [0.2,0.8]) to slicetest, but you don't trigger a 
 short read by doing that (see discussion above).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2983) CliClient print memtable threshold in incorrect order

2011-08-01 Thread Jackson Chung (JIRA)

CliClient print memtable threshold in incorrect order
-

 Key: CASSANDRA-2983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2983
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jackson Chung
Priority: Minor


as a continuation from #2839 looks like it was incorrectly merged into 0.8 as 
well, hence affecting  0.8.2.

for trunk, this is also changed (time is taken out). So I guess format wise, we 
would stick with the fixed format in 0.7.7 per #2839 , which is:

{code}
sessionState.out.printf(  Memtable thresholds: %s/%s/%s (millions of 
ops/minutes/MB)%n,
cf_def.memtable_operations_in_millions, cf_def.memtable_flush_after_mins, 
cf_def.memtable_throughput_in_mb);
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1152967 - /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java

2011-08-01 Thread brandonwilliams

Author: brandonwilliams
Date: Mon Aug  1 23:19:22 2011
New Revision: 1152967

URL: http://svn.apache.org/viewvc?rev=1152967view=rev
Log:
Fix ordering of memtable thresholds in the cli, again.

Modified:

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java?rev=1152967r1=1152966r2=1152967view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java
 Mon Aug  1 23:19:22 2011
@@ -1715,7 +1715,7 @@ public class CliClient
 sessionState.out.printf(  Columns sorted by: %s%s%n, 
cf_def.comparator_type, cf_def.column_type.equals(Super) ? / + 
cf_def.subcomparator_type : );
 sessionState.out.printf(  Row cache size / save period in 
seconds: %s/%s%n, cf_def.row_cache_size, 
cf_def.row_cache_save_period_in_seconds);
 sessionState.out.printf(  Key cache size / save period in 
seconds: %s/%s%n, cf_def.key_cache_size, 
cf_def.key_cache_save_period_in_seconds);
-sessionState.out.printf(  Memtable thresholds: %s/%s/%s 
(millions of ops/MB/minutes)%n,
+sessionState.out.printf(  Memtable thresholds: %s/%s/%s 
(millions of ops/minutes/MB)%n,
 cf_def.memtable_operations_in_millions, 
cf_def.memtable_flush_after_mins, cf_def.memtable_throughput_in_mb);
 sessionState.out.printf(  GC grace seconds: %s%n, 
cf_def.gc_grace_seconds);
 sessionState.out.printf(  Compaction min/max thresholds: 
%s/%s%n, cf_def.min_compaction_threshold, cf_def.max_compaction_threshold);

[jira] [Resolved] (CASSANDRA-2983) CliClient print memtable threshold in incorrect order


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams resolved CASSANDRA-2983.
-

Resolution: Fixed

Fixed in r1152967

 CliClient print memtable threshold in incorrect order
 -

 Key: CASSANDRA-2983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2983
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jackson Chung
Priority: Minor

 as a continuation from #2839 looks like it was incorrectly merged into 0.8 as 
 well, hence affecting  0.8.2.
 for trunk, this is also changed (time is taken out). So I guess format wise, 
 we would stick with the fixed format in 0.7.7 per #2839 , which is:
 {code}
 sessionState.out.printf(  Memtable thresholds: %s/%s/%s (millions of 
 ops/minutes/MB)%n,
 cf_def.memtable_operations_in_millions, cf_def.memtable_flush_after_mins, 
 cf_def.memtable_throughput_in_mb);
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2983) CliClient print memtable threshold in incorrect order


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Williams updated CASSANDRA-2983:


Fix Version/s: 0.8.3

 CliClient print memtable threshold in incorrect order
 -

 Key: CASSANDRA-2983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2983
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Jackson Chung
Priority: Minor
 Fix For: 0.8.3


 as a continuation from #2839 looks like it was incorrectly merged into 0.8 as 
 well, hence affecting  0.8.2.
 for trunk, this is also changed (time is taken out). So I guess format wise, 
 we would stick with the fixed format in 0.7.7 per #2839 , which is:
 {code}
 sessionState.out.printf(  Memtable thresholds: %s/%s/%s (millions of 
 ops/minutes/MB)%n,
 cf_def.memtable_operations_in_millions, cf_def.memtable_flush_after_mins, 
 cf_def.memtable_throughput_in_mb);
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2034:


Attachment: CASSANDRA-2034-trunk-v2.patch

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk-v2.patch, CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled

[
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13075977#comment-13075977
]

Patricio Echague commented on CASSANDRA-2034:
-

v2 patch replaces v1.

Changes in v2:
- It fixes the add-up timeouts by adding a CreationAwareFuture
- Implements the matrix previously discussed

| Hinted Handoff | Consist. Level |
| on | =1 | -- wait for hints. We DO NOT notify the
handler with handler.response() for hints;
| on | ANY | -- wait for hints. Responses count towards
consistency.
| off| =1 | -- DO NOT fire hints. And DO NOT wait for
them to complete.
| off| ANY | -- DO NOT fire hints. And DO NOT wait for
them to complete.

Make Read Repair unnecessary when Hinted Handoff is enabled
---

Attachments: CASSANDRA-2034-trunk-v2.patch, CASSANDRA-2034-trunk.patch

Original Estimate: 8h
Remaining Estimate: 8h

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13075977#comment-13075977
 ] 

Patricio Echague edited comment on CASSANDRA-2034 at 8/2/11 12:38 AM:
--

v2 patch replaces v1.

Changes in v2:
- It fixes the add-up timeouts by adding a CreationAwareFuture
- moves wait method into FBUtilities
- Implements the matrix previously discussed

| Hinted Handoff | Consist. Level |
| on |   =1  | -- wait for hints. We DO NOT notify the 
handler with handler.response() for hints; 
| on |   ANY  | -- wait for hints. Responses count towards 
consistency.
| off|   =1  | -- DO NOT fire hints. And DO NOT wait for 
them to complete.
| off|   ANY  | -- DO NOT fire hints. And DO NOT wait for 
them to complete.

  was (Author: patricioe):
v2 patch replaces v1.

Changes in v2:
- It fixes the add-up timeouts by adding a CreationAwareFuture
- Implements the matrix previously discussed

| Hinted Handoff | Consist. Level |
| on |   =1  | -- wait for hints. We DO NOT notify the 
handler with handler.response() for hints; 
| on |   ANY  | -- wait for hints. Responses count towards 
consistency.
| off|   =1  | -- DO NOT fire hints. And DO NOT wait for 
them to complete.
| off|   ANY  | -- DO NOT fire hints. And DO NOT wait for 
them to complete.
  
 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Patricio Echague
 Fix For: 1.0

 Attachments: CASSANDRA-2034-trunk-v2.patch, CASSANDRA-2034-trunk.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2927) Error truncating a table that is being loaded into


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13076019#comment-13076019
 ] 

Jonathan Ellis commented on CASSANDRA-2927:
---

Any updates?  Did you check the file handle limit theory?

 Error truncating a table that is being loaded into
 --

 Key: CASSANDRA-2927
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2927
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
 Environment: * tip of 0.8 trunk as of July 19th
Reporter: Cathy Daw
Priority: Minor

 * In one window, run a large stress job - 1 rows / 1000 columns
 * In a second window, open the cli and truncate the table being loaded
 * See the following exception on all nodes
  
 {code}
 ERROR 01:14:28,763 Fatal exception in thread 
 Thread[CompactionExecutor:6,1,main]
 java.io.IOError: java.io.IOException: Unable to create compaction marker
   at 
 org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
   at 
 org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
   at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
   at 
 org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
   at 
 org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:979)
   at 
 org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:594)
   at 
 org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:505)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:140)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:106)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 Caused by: java.io.IOException: Unable to create compaction marker
   at 
 org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634)
   ... 13 more
 {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2643) read repair/reconciliation breaks slice based iteration at QUORUM

2011-08-01 Thread Byron Clark (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Byron Clark updated CASSANDRA-2643:
---

Attachment: CASSANDRA-2643-v3.patch

[^CASSANDRA-2643-v3.patch] incorporates Jonathan's suggestions.

 read repair/reconciliation breaks slice based iteration at QUORUM
 -

 Key: CASSANDRA-2643
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2643
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.5
Reporter: Peter Schuller
Assignee: Brandon Williams
Priority: Critical
 Fix For: 1.0

 Attachments: CASSANDRA-2643-poc.patch, CASSANDRA-2643-v2.patch, 
 CASSANDRA-2643-v3.patch, reliable_short_read_0.8.sh, short_read.sh, 
 short_read_0.8.sh, slicetest.py


 In short, I believe iterating over columns is impossible to do reliably with 
 QUORUM due to the way reconciliation works.
 The problem is that the SliceQueryFilter is executing locally when reading on 
 a node, but no attempts seem to be made to consider limits when doing 
 reconciliation and/or read-repair (RowRepairResolver.resolveSuperset() and 
 ColumnFamily.resolve()).
 If a node slices and comes up with 100 columns, and another node slices and 
 comes up with 100 columns, some of which are unique to each side, 
 reconciliation results in  100 columns in the result set. In this case the 
 effect is limited to client gets more than asked for, but the columns still 
 accurately represent the range. This is easily triggered by my test-case.
 In addition to the client receiving too many columns, I believe some of 
 them will not be satisfying the QUORUM consistency level for the same reasons 
 as with deletions (see discussion below).
 Now, there *should* be a problem for tombstones as well, but it's more 
 subtle. Suppose A has:
   1
   2
   3
   4
   5
   6
 and B has:
   1
   del 2
   del 3
   del 4
   5
   6 
 If you now slice 1-6 with count=3 the tombstones from B will reconcile with 
 those from A - fine. So you end up getting 1,5,6 back. This made it a bit 
 difficult to trigger in a test case until I realized what was going on. At 
 first I was hoping to see a short iteration result, which would mean that 
 the process of iterating until you get a short result will cause spurious 
 end of columns and thus make it impossible to iterate correctly.
 So; due to 5-6 existing (and if they didn't, you legitimately reached 
 end-of-columns) we do indeed get a result of size 3 which contains 1,5 and 6. 
 However, only node B would have contributed columns 5 and 6; so there is 
 actually no QUORUM consistency on the co-ordinating node with respect to 
 these columns. If node A and C also had 5 and 6, they would not have been 
 considered.
 Am I wrong?
 In any case; using script I'm about to attach, you can trigger the 
 over-delivery case very easily:
 (0) disable hinted hand-off to avoid that interacting with the test
 (1) start three nodes
 (2) create ks 'test' with rf=3 and cf 'slicetest'
 (3) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (4) stop node A
 (5) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (6) start node A, wait for B and C to consider it up
 (7) ./slicetest.py hostname_of_node_A slice # make A co-ordinator though it 
 doesn't necessarily matter
 You can also pass 'delete' (random deletion of 50% of contents) or 
 'deleterange' (delete all in [0.2,0.8]) to slicetest, but you don't trigger a 
 short read by doing that (see discussion above).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2925) The JDBC Suite should provide a simple DataSource implementation