date:20120726


[ 
https://issues.apache.org/jira/browse/CASSANDRA-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422986#comment-13422986
 ] 

Sylvain Lebresne commented on CASSANDRA-1337:
-

The patch almost move the check for do we have enough rows for the query 
inside the don't use the local path for that range, which is broken (as in 
uselessly inefficient), because it means if we do have enough rows locally, 
we will still check another range.

But probably more importantly, I'm confused on how this patch works.

First, it estimated how much the query of *a range* will likely yield based on 
the *total* number of keys on the node. We should at least divide this by the 
replication factor to have a proper estimate of the number-of-keys-per-range.

Second, it seems to correctly parallelize old-style range_slice (i.e. 
range_slice for thrift without any IndexExpression), but I think it doesn't 
correctly handle neither secondary indexes (which was the main goal of the 
patch I believe), not any kind of range slices when CQL3 is involved:
* For secondary index queries, the number or rows returned doesn't depend at 
all on the number of row keys the node holds (and thus if follows that 
estimating the number of parallel queries to do based on that parameter is 
broken), it depends on how many columns the row for the most selective index 
contains. So for the concurrencyFactor we should 1) figure out which index will 
be used and 2) probably use the estimated mean columns count for that index.
* For CQL3 queries, they use the maxIsColumns parameters (for both traditional 
*and* 2ndary index queries) and so the number of rows returned shouldn't be 
directly compared to maxResults (i.e. if the first row we find has enough 
columns to satisfy maxResults, we're done). In that case, it unfortunately 
become more complicated to predict how much results a query might yield in 
general, because this depends on the column filter. I.e. if the filter is a 
name filter, or an identity slice filter (as in IdentitySliceFilter), we can 
try an estimate (in the latter case, something like maxResults / (estimatedKeys 
* meanColumnsCountPerKey)), but for other kind of slice filters, I don't think 
we can do much estimate. That being said, it might still be worth using the 
estimate in those two cases, because at least for 2ndary index query, the 
column filter will likely be very often an identity slice. And in that 
identity slice case, for 2ndary index and CQL3, the estimation should be 
something like maxResults / (meanColumnCountPerKey(index used) * 
meanColumnCountPerKey(parent cf)).

I'll note however that in both case, the meanColumnsCount is not necessary the 
perfect estimate to use, as it pretty much imply that half the time we will 
query one more range than is necessary. Instead we could either use the 
maxColumnsCount (if we really want to be conservative) or add some fudge factor 
to the mean. Fudge factor that may maybe be based on the difference between the 
min, mean and max columns count estimates.

 parallelize fetching rows for low-cardinality indexes
 -

 Key: CASSANDRA-1337
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1337
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jonathan Ellis
Assignee: David Alves
Priority: Minor
 Fix For: 1.2

 Attachments: 
 0001-CASSANDRA-1337-scan-concurrently-depending-on-num-rows.txt, 
 CASSANDRA-1337.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 currently, we read the indexed rows from the first node (in partitioner 
 order); if that does not have enough matching rows, we read the rows from the 
 next, and so forth.
 we should use the statistics fom CASSANDRA-1155 to query multiple nodes in 
 parallel, such that we have a high chance of getting enough rows w/o having 
 to do another round of queries (but, if our estimate is incorrect, we do need 
 to loop and do more rounds until we have enough data or we have fetched from 
 each node).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-4436) Counters in columns don't preserve correct values after cluster restart

[
https://issues.apache.org/jira/browse/CASSANDRA-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-4436:

Attachment: 4436-1.1-2.txt
4436-1.0-2.txt

You're completely right, I'm still thinking too much in terms of
SizeTieredCompaction.

Updated patches to use a Set.

Counters in columns don't preserve correct values after cluster restart
---

Key: CASSANDRA-4436
URL: https://issues.apache.org/jira/browse/CASSANDRA-4436
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.10
Reporter: Peter Velas
Assignee: Sylvain Lebresne
Fix For: 1.1.3

Attachments: 4436-1.0-2.txt, 4436-1.0-2.txt, 4436-1.0.txt,
4436-1.1-2.txt, 4436-1.1-2.txt, 4436-1.1.txt, increments.cql.gz

Similar to #3821. but affecting normal columns.
Set up a 2-node cluster with rf=2.
1. Create a counter column family and increment a 100 keys in loop 5000
times.
2. Then make a rolling restart to cluster.
3. Again increment another 5000 times.
4. Make a rolling restart to cluster.
5. Again increment another 5000 times.
6. Make a rolling restart to cluster.
After step 6 we were able to reproduce bug with bad counter values.
Expected values were 15 000. Values returned from cluster are higher then
15000 + some random number.
Rolling restarts are done with nodetool drain. Always waiting until second
node discover its down then kill java process.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

git commit: fix String.format parameter type for enum value

Updated Branches:
  refs/heads/trunk d96e813e0 - 732d83b75


fix String.format parameter type for enum value


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/732d83b7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/732d83b7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/732d83b7

Branch: refs/heads/trunk
Commit: 732d83b7581b89394bd28a8c03124bff733efe7c
Parents: d96e813
Author: Dave Brosius dbros...@apache.org
Authored: Wed Jul 25 23:46:01 2012 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Wed Jul 25 23:46:01 2012 -0400

--
 .../apache/cassandra/transport/OptionCodec.java|2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/732d83b7/src/java/org/apache/cassandra/transport/OptionCodec.java
--
diff --git a/src/java/org/apache/cassandra/transport/OptionCodec.java 
b/src/java/org/apache/cassandra/transport/OptionCodec.java
index 2898300..71d3193 100644
--- a/src/java/org/apache/cassandra/transport/OptionCodec.java
+++ b/src/java/org/apache/cassandra/transport/OptionCodec.java
@@ -77,7 +77,7 @@ public class OptionCodecT extends EnumT  
OptionCodec.CodecableT
 T opt = fromId(body.readUnsignedShort());
 Object value = opt.readValue(body);
 if (options.containsKey(opt))
-throw new ProtocolException(String.format(Duplicate option %d 
in message, opt));
+throw new ProtocolException(String.format(Duplicate option %s 
in message, opt.name()));
 options.put(opt, value);
 }
 return options;

[jira] [Updated] (CASSANDRA-1337) parallelize fetching rows for low-cardinality indexes


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-1337:


Priority: Major  (was: Minor)

(changing the priority to major because basically I think that currently it 
does more harm than good, and we should either fix it before release or revert 
it)

 parallelize fetching rows for low-cardinality indexes
 -

 Key: CASSANDRA-1337
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1337
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jonathan Ellis
Assignee: David Alves
 Fix For: 1.2

 Attachments: 
 0001-CASSANDRA-1337-scan-concurrently-depending-on-num-rows.txt, 
 CASSANDRA-1337.patch

   Original Estimate: 8h
  Remaining Estimate: 8h

 currently, we read the indexed rows from the first node (in partitioner 
 order); if that does not have enough matching rows, we read the rows from the 
 next, and so forth.
 we should use the statistics fom CASSANDRA-1155 to query multiple nodes in 
 parallel, such that we have a high chance of getting enough rows w/o having 
 to do another round of queries (but, if our estimate is incorrect, we do need 
 to loop and do more rounds until we have enough data or we have fetched from 
 each node).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

git commit: fix String.format parameter type for UUID

Updated Branches:
  refs/heads/trunk 732d83b75 - fae7a23b5


fix String.format parameter type for UUID


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fae7a23b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fae7a23b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fae7a23b

Branch: refs/heads/trunk
Commit: fae7a23b5f1ff2d3335ddbfe56c1c8973c8803fe
Parents: 732d83b
Author: Dave Brosius dbros...@apache.org
Authored: Wed Jul 25 23:54:25 2012 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Wed Jul 25 23:54:25 2012 -0400

--
 .../org/apache/cassandra/cache/RowCacheKey.java|2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fae7a23b/src/java/org/apache/cassandra/cache/RowCacheKey.java
--
diff --git a/src/java/org/apache/cassandra/cache/RowCacheKey.java 
b/src/java/org/apache/cassandra/cache/RowCacheKey.java
index 2477a06..f6c743f 100644
--- a/src/java/org/apache/cassandra/cache/RowCacheKey.java
+++ b/src/java/org/apache/cassandra/cache/RowCacheKey.java
@@ -76,6 +76,6 @@ public class RowCacheKey implements CacheKey, 
ComparableRowCacheKey
 @Override
 public String toString()
 {
-return String.format(RowCacheKey(cfId:%d, key:%s), cfId, 
Arrays.toString(key));
+return String.format(RowCacheKey(cfId:%s, key:%s), cfId, 
Arrays.toString(key));
 }
 }

[2/2] cleanup unused import warnings, and trailing whitespace

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f650d3e8/src/java/org/apache/cassandra/service/MigrationTask.java
--
diff --git a/src/java/org/apache/cassandra/service/MigrationTask.java 
b/src/java/org/apache/cassandra/service/MigrationTask.java
index 7124f16..70b917d 100644
--- a/src/java/org/apache/cassandra/service/MigrationTask.java
+++ b/src/java/org/apache/cassandra/service/MigrationTask.java
@@ -20,14 +20,11 @@ package org.apache.cassandra.service;
 import java.io.IOException;
 import java.net.InetAddress;
 import java.util.Collection;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.config.ConfigurationException;
-import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.DefsTable;
 import org.apache.cassandra.db.RowMutation;
 import org.apache.cassandra.gms.FailureDetector;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f650d3e8/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java 
b/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java
index 2449d75..e3b1d79 100644
--- a/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java
+++ b/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java
@@ -28,8 +28,6 @@ import org.apache.cassandra.db.RangeSliceCommand;
 import org.apache.cassandra.db.RangeSliceReply;
 import org.apache.cassandra.db.Row;
 import org.apache.cassandra.db.Table;
-import org.apache.cassandra.db.filter.QueryFilter;
-import org.apache.cassandra.db.filter.IFilter;
 import org.apache.cassandra.net.IVerbHandler;
 import org.apache.cassandra.net.MessageIn;
 import org.apache.cassandra.net.MessagingService;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f650d3e8/src/java/org/apache/cassandra/service/RowDigestResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/RowDigestResolver.java 
b/src/java/org/apache/cassandra/service/RowDigestResolver.java
index 056f2fb..e0e262b 100644
--- a/src/java/org/apache/cassandra/service/RowDigestResolver.java
+++ b/src/java/org/apache/cassandra/service/RowDigestResolver.java
@@ -19,7 +19,6 @@ package org.apache.cassandra.service;
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
-import java.util.Map;
 
 import org.apache.cassandra.db.ColumnFamily;
 import org.apache.cassandra.db.ReadResponse;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f650d3e8/src/java/org/apache/cassandra/service/RowRepairResolver.java
--
diff --git a/src/java/org/apache/cassandra/service/RowRepairResolver.java 
b/src/java/org/apache/cassandra/service/RowRepairResolver.java
index cb5ba32..5618cf0 100644
--- a/src/java/org/apache/cassandra/service/RowRepairResolver.java
+++ b/src/java/org/apache/cassandra/service/RowRepairResolver.java
@@ -23,7 +23,6 @@ import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
-import java.util.Map;
 
 import com.google.common.collect.Iterables;
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f650d3e8/src/java/org/apache/cassandra/service/StorageServiceMBean.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageServiceMBean.java 
b/src/java/org/apache/cassandra/service/StorageServiceMBean.java
index 2224180..0c4bfa3 100644
--- a/src/java/org/apache/cassandra/service/StorageServiceMBean.java
+++ b/src/java/org/apache/cassandra/service/StorageServiceMBean.java
@@ -21,7 +21,6 @@ import java.io.IOException;
 import java.net.InetAddress;
 import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
-import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ExecutionException;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f650d3e8/src/java/org/apache/cassandra/streaming/FileStreamTask.java
--
diff --git a/src/java/org/apache/cassandra/streaming/FileStreamTask.java 
b/src/java/org/apache/cassandra/streaming/FileStreamTask.java
index b557573..ba7d9f3 100644
--- a/src/java/org/apache/cassandra/streaming/FileStreamTask.java
+++ b/src/java/org/apache/cassandra/streaming/FileStreamTask.java
@@ -27,7 +27,6 @@ import org.slf4j.LoggerFactory;
 
 import com.ning.compress.lzf.LZFOutputStream;
 import org.apache.cassandra.config.DatabaseDescriptor;
-import org.apache.cassandra.gms.Gossiper;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.io.util.RandomAccessReader;
 import

[jira] [Commented] (CASSANDRA-4411) Assertion with LCS compaction

2012-07-26 Thread Mina Naguib (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423108#comment-13423108
 ] 

Mina Naguib commented on CASSANDRA-4411:


Quick follow-up

All the problematic nodes have been offline scrubbed (successfully using 
--manifest-check to speed things up).  There are no more compaction errors / 
pending compactions.

Like Anton, I'm a bit weary and keeping an eye on things - but so far so good.

On a tangent, it occurred to me that the amount of time it takes to run ( 
sstablescrub --manifest-check ) is mostly reading the sstables - the check 
itself and demoting the bad sstables to L0 appears very cheap - would it be a 
good idea to perform that check automatically on cassandra startup (after the 
sstables have been read) ?  It *may* be a quick fix for 1.1.3 to help people 
out who have been bitten by this but don't know it yet.



 Assertion with LCS compaction
 -

 Key: CASSANDRA-4411
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4411
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2
Reporter: Anton Winter
Assignee: Sylvain Lebresne
 Fix For: 1.1.3

 Attachments: 0001-Add-debugging-info-for-LCS.txt, 4411-followup.txt, 
 4411.txt, assertion-w-more-debugging-info-omid.log, 
 assertion.moreinfo.system.log, system.log


 As instructed in CASSANDRA-4321 I have raised this issue as a continuation of 
 that issue as it appears the problem still exists.
 I have repeatedly run sstablescrub across all my nodes after the 1.1.2 
 upgrade until sstablescrub shows no errors.  The exceptions described in 
 CASSANDRA-4321 do not occur as frequently now but the integrity check still 
 throws exceptions on a number of nodes.  Once those exceptions occur 
 compactionstats shows a large number of pending tasks with no progression 
 afterwards.
 {code}
 ERROR [CompactionExecutor:150] 2012-07-05 04:26:15,570 
 AbstractCassandraDaemon.java (line 134) Exception in thread 
 Thread[CompactionExecutor:150,1,main]
 java.lang.AssertionError
 at 
 org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:214)
 at 
 org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:158)
 at 
 org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:531)
 at 
 org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:254)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:978)
 at 
 org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
 at 
 org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
 at 
 org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:636)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4436) Counters in columns don't preserve correct values after cluster restart

[
https://issues.apache.org/jira/browse/CASSANDRA-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423110#comment-13423110
]

Jonathan Ellis commented on CASSANDRA-4436:
---

Counters in columns don't preserve correct values after cluster restart
---

Attachments: 4436-1.0-2.txt, 4436-1.0-2.txt, 4436-1.0.txt,
4436-1.1-2.txt, 4436-1.1-2.txt, 4436-1.1.txt, increments.cql.gz

git commit: Ensure compacted sstables are never used

Updated Branches:
  refs/heads/cassandra-1.1 cc0be1b40 - 6a6b7ec1f


Ensure compacted sstables are never used

patch by slebresne; reviewed by jbellis for CASSANDRA-4436


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6a6b7ec1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6a6b7ec1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6a6b7ec1

Branch: refs/heads/cassandra-1.1
Commit: 6a6b7ec1f00d8084fb29379111a798c09c35e6d6
Parents: cc0be1b
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Wed Jul 18 19:54:20 2012 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Thu Jul 26 17:46:19 2012 +0200

--
 CHANGES.txt|1 +
 .../org/apache/cassandra/db/ColumnFamilyStore.java |   32 ++--
 .../org/apache/cassandra/io/sstable/Component.java |2 +
 .../apache/cassandra/io/sstable/Descriptor.java|   10 +++-
 .../org/apache/cassandra/io/sstable/SSTable.java   |2 +
 .../cassandra/io/sstable/SSTableMetadata.java  |   35 +
 .../apache/cassandra/io/sstable/SSTableReader.java |   39 +--
 7 files changed, 80 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a6b7ec1/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 169f66d..84db73d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -19,6 +19,7 @@
  * fix Can't Modify Index Name problem on CF update (CASSANDRA-4439)
  * Fix assertion error in getOverlappingSSTables during repair (CASSANDRA-4456)
  * fix nodetool's setcompactionthreshold command (CASSANDRA-4455)
+ * Ensure compacted files are never used, to avoid counter overcount 
(CASSANDRA-4436)
 Merged from 1.0:
  * allow dropping columns shadowed by not-yet-expired supercolumn or row
tombstones in PrecompactedRow (CASSANDRA-4396)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a6b7ec1/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index b93adc1..a39530a 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -234,8 +234,23 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 if (loadSSTables)
 {
-Directories.SSTableLister sstables = 
directories.sstableLister().skipCompacted(true).skipTemporary(true);
-
data.addInitialSSTables(SSTableReader.batchOpen(sstables.list().entrySet(), 
savedKeys, data, metadata, this.partitioner));
+Directories.SSTableLister sstableFiles = 
directories.sstableLister().skipCompacted(true).skipTemporary(true);
+CollectionSSTableReader sstables = 
SSTableReader.batchOpen(sstableFiles.list().entrySet(), savedKeys, data, 
metadata, this.partitioner);
+
+// Filter non-compacted sstables, remove compacted ones
+SetInteger compactedSSTables = new HashSetInteger();
+for (SSTableReader sstable : sstables)
+compactedSSTables.addAll(sstable.getAncestors());
+
+SetSSTableReader liveSSTables = new HashSetSSTableReader();
+for (SSTableReader sstable : sstables)
+{
+if (compactedSSTables.contains(sstable.descriptor.generation))
+sstable.releaseReference(); // this amount to deleting the 
sstable
+else
+liveSSTables.add(sstable);
+}
+data.addInitialSSTables(liveSSTables);
 }
 
 // compaction strategy should be created after the CFS has been 
prepared
@@ -492,7 +507,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 SSTableReader reader;
 try
 {
-reader = SSTableReader.open(newDescriptor, entry.getValue(), 
Collections.DecoratedKeyemptySet(), data, metadata, partitioner);
+reader = SSTableReader.open(newDescriptor, entry.getValue(), 
Collections.DecoratedKeyemptySet(), metadata, partitioner);
 }
 catch (IOException e)
 {
@@ -1969,9 +1984,18 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 ReplayPosition rp = ReplayPosition.getReplayPosition(sstables);
 SSTableMetadata.Collector sstableMetadataCollector = 
SSTableMetadata.createCollector().replayPosition(rp);
 
-// get the max timestamp of the precompacted sstables
+// Get the max timestamp of the precompacted sstables
+// and adds generation of live ancestors

[2/2] git commit: Ensure compacted sstables are never used

Ensure compacted sstables are never used

patch by slebresne; reviewed by jbellis for CASSANDRA-4436


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6a6b7ec1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6a6b7ec1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6a6b7ec1

Branch: refs/heads/trunk
Commit: 6a6b7ec1f00d8084fb29379111a798c09c35e6d6
Parents: cc0be1b
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Wed Jul 18 19:54:20 2012 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Thu Jul 26 17:46:19 2012 +0200

--
 CHANGES.txt|1 +
 .../org/apache/cassandra/db/ColumnFamilyStore.java |   32 ++--
 .../org/apache/cassandra/io/sstable/Component.java |2 +
 .../apache/cassandra/io/sstable/Descriptor.java|   10 +++-
 .../org/apache/cassandra/io/sstable/SSTable.java   |2 +
 .../cassandra/io/sstable/SSTableMetadata.java  |   35 +
 .../apache/cassandra/io/sstable/SSTableReader.java |   39 +--
 7 files changed, 80 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a6b7ec1/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 169f66d..84db73d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -19,6 +19,7 @@
  * fix Can't Modify Index Name problem on CF update (CASSANDRA-4439)
  * Fix assertion error in getOverlappingSSTables during repair (CASSANDRA-4456)
  * fix nodetool's setcompactionthreshold command (CASSANDRA-4455)
+ * Ensure compacted files are never used, to avoid counter overcount 
(CASSANDRA-4436)
 Merged from 1.0:
  * allow dropping columns shadowed by not-yet-expired supercolumn or row
tombstones in PrecompactedRow (CASSANDRA-4396)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6a6b7ec1/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index b93adc1..a39530a 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -234,8 +234,23 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 
 if (loadSSTables)
 {
-Directories.SSTableLister sstables = 
directories.sstableLister().skipCompacted(true).skipTemporary(true);
-
data.addInitialSSTables(SSTableReader.batchOpen(sstables.list().entrySet(), 
savedKeys, data, metadata, this.partitioner));
+Directories.SSTableLister sstableFiles = 
directories.sstableLister().skipCompacted(true).skipTemporary(true);
+CollectionSSTableReader sstables = 
SSTableReader.batchOpen(sstableFiles.list().entrySet(), savedKeys, data, 
metadata, this.partitioner);
+
+// Filter non-compacted sstables, remove compacted ones
+SetInteger compactedSSTables = new HashSetInteger();
+for (SSTableReader sstable : sstables)
+compactedSSTables.addAll(sstable.getAncestors());
+
+SetSSTableReader liveSSTables = new HashSetSSTableReader();
+for (SSTableReader sstable : sstables)
+{
+if (compactedSSTables.contains(sstable.descriptor.generation))
+sstable.releaseReference(); // this amount to deleting the 
sstable
+else
+liveSSTables.add(sstable);
+}
+data.addInitialSSTables(liveSSTables);
 }
 
 // compaction strategy should be created after the CFS has been 
prepared
@@ -492,7 +507,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 SSTableReader reader;
 try
 {
-reader = SSTableReader.open(newDescriptor, entry.getValue(), 
Collections.DecoratedKeyemptySet(), data, metadata, partitioner);
+reader = SSTableReader.open(newDescriptor, entry.getValue(), 
Collections.DecoratedKeyemptySet(), metadata, partitioner);
 }
 catch (IOException e)
 {
@@ -1969,9 +1984,18 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 ReplayPosition rp = ReplayPosition.getReplayPosition(sstables);
 SSTableMetadata.Collector sstableMetadataCollector = 
SSTableMetadata.createCollector().replayPosition(rp);
 
-// get the max timestamp of the precompacted sstables
+// Get the max timestamp of the precompacted sstables
+// and adds generation of live ancestors
 for (SSTableReader sstable : sstables)
+{

[1/2] git commit: Merge branch 'cassandra-1.1' into trunk

Updated Branches:
  refs/heads/trunk f650d3e82 - 111f6c939


Merge branch 'cassandra-1.1' into trunk

Conflicts:
src/java/org/apache/cassandra/db/ColumnFamilyStore.java
src/java/org/apache/cassandra/io/sstable/Descriptor.java
src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java
src/java/org/apache/cassandra/io/sstable/SSTableReader.java


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/111f6c93
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/111f6c93
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/111f6c93

Branch: refs/heads/trunk
Commit: 111f6c9392766205823e326422f3dffae826cf97
Parents: f650d3e 6a6b7ec
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Thu Jul 26 18:04:14 2012 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Thu Jul 26 18:04:14 2012 +0200

--
 CHANGES.txt|1 +
 .../org/apache/cassandra/db/ColumnFamilyStore.java |   34 +++--
 src/java/org/apache/cassandra/db/Directories.java  |   11 
 .../org/apache/cassandra/io/sstable/Component.java |2 +
 .../apache/cassandra/io/sstable/Descriptor.java|8 +++
 .../org/apache/cassandra/io/sstable/SSTable.java   |2 +
 .../cassandra/io/sstable/SSTableMetadata.java  |   33 +---
 .../apache/cassandra/io/sstable/SSTableReader.java |   40 ---
 .../apache/cassandra/tools/StandaloneScrubber.java |2 +-
 .../org/apache/cassandra/db/DirectoriesTest.java   |   21 ++--
 10 files changed, 84 insertions(+), 70 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/111f6c93/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/111f6c93/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 7692e59,a39530a..695e480
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -229,13 -231,28 +229,28 @@@ public class ColumnFamilyStore implemen
  
  if (loadSSTables)
  {
- Directories.SSTableLister sstables = 
directories.sstableLister().skipCompacted(true).skipTemporary(true);
- 
data.addInitialSSTables(SSTableReader.batchOpen(sstables.list().entrySet(), 
data, metadata, this.partitioner));
 -Directories.SSTableLister sstableFiles = 
directories.sstableLister().skipCompacted(true).skipTemporary(true);
 -CollectionSSTableReader sstables = 
SSTableReader.batchOpen(sstableFiles.list().entrySet(), savedKeys, data, 
metadata, this.partitioner);
++Directories.SSTableLister sstableFiles = 
directories.sstableLister().skipTemporary(true);
++CollectionSSTableReader sstables = 
SSTableReader.batchOpen(sstableFiles.list().entrySet(), data, metadata, 
this.partitioner);
+ 
+ // Filter non-compacted sstables, remove compacted ones
+ SetInteger compactedSSTables = new HashSetInteger();
+ for (SSTableReader sstable : sstables)
+ compactedSSTables.addAll(sstable.getAncestors());
+ 
+ SetSSTableReader liveSSTables = new HashSetSSTableReader();
+ for (SSTableReader sstable : sstables)
+ {
+ if (compactedSSTables.contains(sstable.descriptor.generation))
+ sstable.releaseReference(); // this amount to deleting 
the sstable
+ else
+ liveSSTables.add(sstable);
+ }
+ data.addInitialSSTables(liveSSTables);
  }
  
 +if (caching == Caching.ALL || caching == Caching.KEYS_ONLY)
 +CacheService.instance.keyCache.loadSaved(this);
 +
  // compaction strategy should be created after the CFS has been 
prepared
  this.compactionStrategy = 
metadata.createCompactionStrategyInstance(this);
  
@@@ -451,7 -480,7 +466,7 @@@
  currentDescriptors.add(sstable.descriptor);
  SetSSTableReader newSSTables = new HashSetSSTableReader();
  
--Directories.SSTableLister lister = 
directories.sstableLister().skipCompacted(true).skipTemporary(true);
++Directories.SSTableLister lister = 
directories.sstableLister().skipTemporary(true);
  for (Map.EntryDescriptor, SetComponent entry : 
lister.list().entrySet())
  {
  Descriptor descriptor = entry.getKey();
@@@ -478,7 -507,7 +493,7 @@@
  SSTableReader reader;
  try
  {
- reader = SSTableReader.open(newDescriptor, entry.getValue(), 
data, metadata,

git commit: Fix comments

Updated Branches:
  refs/heads/cassandra-1.1 6a6b7ec1f - bb37a0f25


Fix comments


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bb37a0f2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bb37a0f2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bb37a0f2

Branch: refs/heads/cassandra-1.1
Commit: bb37a0f250481604c333fabd30d66ae19dcbef44
Parents: 6a6b7ec
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Thu Jul 26 18:05:21 2012 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Thu Jul 26 18:05:21 2012 +0200

--
 .../apache/cassandra/io/sstable/Descriptor.java|2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bb37a0f2/src/java/org/apache/cassandra/io/sstable/Descriptor.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/Descriptor.java 
b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
index 07bd6e8..d35865c 100644
--- a/src/java/org/apache/cassandra/io/sstable/Descriptor.java
+++ b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
@@ -58,7 +58,7 @@ public class Descriptor
 // hb (1.0.3): records compression ration in metadata component
 // hc (1.0.4): records partitioner in metadata component
 // hd (1.0.10): includes row tombstones in maxtimestamp
-// he (1.0.11): includes row tombstones in maxtimestamp
+// he (1.1.3): includes ancestors generation in metadata component
 public static final String CURRENT_VERSION = he;
 
 public final File directory;

[2/2] git commit: Fix comments

Fix comments


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bb37a0f2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bb37a0f2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bb37a0f2

Branch: refs/heads/trunk
Commit: bb37a0f250481604c333fabd30d66ae19dcbef44
Parents: 6a6b7ec
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Thu Jul 26 18:05:21 2012 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Thu Jul 26 18:05:21 2012 +0200

--
 .../apache/cassandra/io/sstable/Descriptor.java|2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bb37a0f2/src/java/org/apache/cassandra/io/sstable/Descriptor.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/Descriptor.java 
b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
index 07bd6e8..d35865c 100644
--- a/src/java/org/apache/cassandra/io/sstable/Descriptor.java
+++ b/src/java/org/apache/cassandra/io/sstable/Descriptor.java
@@ -58,7 +58,7 @@ public class Descriptor
 // hb (1.0.3): records compression ration in metadata component
 // hc (1.0.4): records partitioner in metadata component
 // hd (1.0.10): includes row tombstones in maxtimestamp
-// he (1.0.11): includes row tombstones in maxtimestamp
+// he (1.1.3): includes ancestors generation in metadata component
 public static final String CURRENT_VERSION = he;
 
 public final File directory;

[1/2] git commit: Merge branch 'cassandra-1.1' into trunk

Updated Branches:
  refs/heads/trunk 111f6c939 - 1507ce8cd


Merge branch 'cassandra-1.1' into trunk

Conflicts:
src/java/org/apache/cassandra/io/sstable/Descriptor.java


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1507ce8c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1507ce8c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1507ce8c

Branch: refs/heads/trunk
Commit: 1507ce8cd4c5c9896a4923d9e0a74d85b6a83184
Parents: 111f6c9 bb37a0f
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Thu Jul 26 18:05:54 2012 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Thu Jul 26 18:05:54 2012 +0200

--

--

[jira] [Updated] (CASSANDRA-4021) CFS.scrubDataDirectories tries to delete nonexistent orphans


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-4021:
--

Fix Version/s: (was: 1.0.11)

 CFS.scrubDataDirectories tries to delete nonexistent orphans
 

 Key: CASSANDRA-4021
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4021
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.7 beta 2
Reporter: Brandon Williams
Assignee: Brandon Williams
Priority: Minor
  Labels: datastax_qa
 Attachments: 4021.txt


 The check only looks for a missing data file, then deletes all other 
 components, however it's possible for the data file and another component to 
 be missing, causing an error:
 {noformat}
  WARN 17:19:28,765 Removing orphans for 
 /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492:
  [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db]
 ERROR 17:19:28,766 Exception encountered during startup
 java.lang.AssertionError: attempted to delete non-existing file 
 system-HintsColumnFamily-hd-24492-Index.db
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357)
 at 
 org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
 at 
 org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352)
 at 
 org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105)
 java.lang.AssertionError: attempted to delete non-existing file 
 system-HintsColumnFamily-hd-24492-Index.db
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49)
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357)
 at 
 org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
 at 
 org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352)
 at 
 org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105)
 Exception encountered during startup: attempted to delete non-existing file 
 system-HintsColumnFamily-hd-24492-Index.db
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4462) upgradesstables strips active data from sstables

[
https://issues.apache.org/jira/browse/CASSANDRA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423231#comment-13423231
]

Sylvain Lebresne commented on CASSANDRA-4462:
-

Best way to make sure would be to test with the patch :)

But since you have ttl, I would say that yes, there is a good chance that's
related.

upgradesstables strips active data from sstables

Key: CASSANDRA-4462
URL: https://issues.apache.org/jira/browse/CASSANDRA-4462
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.2
Environment: Ubuntu 11.04 64-bit
Reporter: Mike Heffner
Fix For: 1.1.3

Attachments: 4462.txt

From the discussion here:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201207.mbox/%3CCAOac0GCtyDqS6ocuHOuQqre4re5wKj3o-ZpUZGkGsjCHzDVbTA%40mail.gmail.com%3E
We are trying to migrate a 0.8.8 cluster to 1.1.2 by migrating the sstables
from the 0.8.8 ring to a parallel 1.1.2 ring. However, every time we run the
`nodetool upgradesstables` step we find it removes active data from our CFs
-- leading to lost data in our application.
The steps we took were:
1. Bring up a 1.1.2 ring in the same AZ/data center configuration with
tokens matching the corresponding nodes in the 0.8.8 ring.
2. Create the same keyspace on 1.1.2.
3. Create each CF in the keyspace on 1.1.2.
4. Flush each node of the 0.8.8 ring.
5. Rsync each non-compacted sstable from 0.8.8 to the corresponding node in
1.1.2.
6. Move each 0.8.8 sstable into the 1.1.2 directory structure by renaming the
file to the /cassandra/data/keyspace/cf/keyspace-cf... format. For
example, for the keyspace Metrics and CF epochs_60 we get:
cassandra/data/Metrics/epochs_60/Metrics-epochs_60-g-941-Data.db.
7. On each 1.1.2 node run `nodetool -h localhost refresh Metrics CF` for
each CF in the keyspace. We notice that storage load jumps accordingly.
8. On each 1.1.2 node run `nodetool -h localhost upgradesstables`.
Afterwards we would test the validity of the data by comparing it with data
from the original 0.8.8 ring. After an upgradesstables command the data was
always incorrect.
With further testing we found that we could successfully use scrub to convert
our sstables without data loss. However, any invocation of upgradesstables
causes active data to be culled from the sstables:
INFO [CompactionExecutor:4] 2012-07-24 04:27:36,837 CompactionTask.java
(line 109) Compacting
[SSTableReader(path='/raid0/cassandra/data/Metrics/metrics_900/Metrics-metrics_900-hd-51-Data.db')]
INFO [CompactionExecutor:4] 2012-07-24 04:27:51,090 CompactionTask.java
(line 221) Compacted to
[/raid0/cassandra/data/Metrics/metrics_900/Metrics-metrics_900-hd-58-Data.db,].
60,449,155 to 2,578,102 (~4% of original) bytes for 4,002 keys at
0.172562MB/s. Time: 14,248ms.
These are the steps we've tried:
WORKS refresh - scrub
WORKS refresh - scrub - major compaction
WORKS refresh - scrub - cleanup
WORKS refresh - scrub - repair
FAILS refresh - upgradesstables
FAILS refresh - scrub - upgradesstables
FAILS refresh - scrub - repair - upgradesstables
FAILS refresh - scrub - major compaction - upgradesstables
We have fewer than 143 million row keys in the CFs we're testing and none
of the *-Filter.db files are 10MB, so I don't believe this is our
problem: https://issues.apache.org/jira/browse/CASSANDRA-3820
The keyspace is defined as:
Keyspace: Metrics:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [us-east:3]
And the column family that we tested with is defined as:
ColumnFamily: metrics_900
Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type
Default column value validator:
org.apache.cassandra.db.marshal.BytesType
Columns sorted by:
org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type)
GC grace seconds: 0
Compaction min/max thresholds: 4/32
Read repair chance: 0.1
DC Local Read repair chance: 0.0
Replicate on write: true
Caching: KEYS_ONLY
Bloom Filter FP chance: default
Built indexes: []
Compaction Strategy:
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
Compression Options:
sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
All rows have a TTL of 30 days and a gc_grace=0 so it's possible that a small
number of older columns would be removed during a
compaction/scrub/upgradesstables step. However, the majority should still be
kept as

[jira] [Updated] (CASSANDRA-3841) long-test timing out


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-3841:
--

Fix Version/s: (was: 1.1.3)
   1.1.4
 Assignee: David Alves

 long-test timing out
 

 Key: CASSANDRA-3841
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3841
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.1.0
Reporter: Michael Allen
Assignee: David Alves
Priority: Minor
 Fix For: 1.1.4


 [junit] -  ---
 [junit] Testsuite: 
 org.apache.cassandra.db.compaction.LongCompactionSpeedTest
 [junit] Testsuite: 
 org.apache.cassandra.db.compaction.LongCompactionSpeedTest
 [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
 [junit] 
 [junit] Testcase: 
 org.apache.cassandra.db.compaction.LongCompactionSpeedTest:BeforeFirstTest:   
   Caused an ERROR
 [junit] Timeout occurred. Please note the time in the report does not 
 reflect the time until the timeout.
 [junit] junit.framework.AssertionFailedError: Timeout occurred. Please 
 note the time in the report does not reflect the time until the timeout.
 [junit] 
 [junit] 
 [junit] Test org.apache.cassandra.db.compaction.LongCompactionSpeedTest 
 FAILED (timeout)
 [junit] Testsuite: org.apache.cassandra.utils.LongBloomFilterTest
 [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 64.536 sec
 [junit] 
 [junit] Testsuite: org.apache.cassandra.utils.LongLegacyBloomFilterTest
 [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 41.104 sec
 [junit] 
 BUILD FAILED
 /Users/mallen/dstax/repos/git/cassandra/build.xml:1113: The following error 
 occurred while executing this line:
 /Users/mallen/dstax/repos/git/cassandra/build.xml:1036: Some long test(s) 
 failed.
 Total time: 63 minutes 9 seconds

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-4410) BulkOutputFormat and CompositeColumn results in java.lang.AssertionError: Added column does not sort as the last column


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-4410:
--

Fix Version/s: (was: 1.1.3)
   1.1.4

 BulkOutputFormat and CompositeColumn results in java.lang.AssertionError: 
 Added column does not sort as the last column
 ---

 Key: CASSANDRA-4410
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4410
 Project: Cassandra
  Issue Type: Bug
  Components: Hadoop
Affects Versions: 1.1.1
 Environment: hadoop 0.20
 cassandra 1.1.1
 cql 3.0.0
Reporter: Henning Kropp
Assignee: Pavel Yaskevich
Priority: Minor
  Labels: bulkloader, compositeColumns, cql3, hadoop
 Fix For: 1.1.4

 Attachments: CompositeColumnsBulkOutputDriver.java, 
 CompositeColumnsBulkOutputMapper.java, 
 CompositeColumnsBulkOutputReducer.java, HectorInsertTest.java, input.csv, 
 keyspace.cli, pom.xml, schema.cql, schema_hector.cql


 Using the org.apache.cassandra.hadoop.BulkOutputFormat.class with 
 CompositeColumns results in the following AsertionError during a Select query 
 with WHERE ID='id' or using an ID in any way in the where clause.
 ERROR [ReadStage:34] 2012-07-04 14:33:27,376 AbstractCassandraDaemon.java 
 (line 134) Exception in thread Thread[ReadStage:34,5,main]
 java.lang.AssertionError: Added column does not sort as the last column
   at 
 org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:130)
   at 
 org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:107)
   at 
 org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:102)
   at 
 org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:141)
   at 
 org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:139)
   at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:283)
   at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
   at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1321)
   at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1183)
   at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1118)
   at org.apache.cassandra.db.Table.getRow(Table.java:374)
   at 
 org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
   at 
 org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:816)
   at 
 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1250)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4462) upgradesstables strips active data from sstables

2012-07-26 Thread Mike Heffner (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423249#comment-13423249
]

Mike Heffner commented on CASSANDRA-4462:
-

Fair enough, though we are not quite equipped to test patch sets yet so it
might take awhile.

upgradesstables strips active data from sstables

Attachments: 4462.txt

--
This message is

[jira] [Commented] (CASSANDRA-4462) upgradesstables strips active data from sstables

[
https://issues.apache.org/jira/browse/CASSANDRA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423252#comment-13423252
]

Jonathan Ellis commented on CASSANDRA-4462:
---

+1 on the basic fix, but I'm not really convinced that GC_ALL and NO_GC are
improvements since we don't really get any extra typesafety from it. (Maybe
switching to an enum for ALL, NONE, and CURRENT_TIME enum would be okay though?
But that's out of scope here.)

upgradesstables strips active data from sstables

Key: CASSANDRA-4462
URL: https://issues.apache.org/jira/browse/CASSANDRA-4462
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.4
Environment: Ubuntu 11.04 64-bit
Reporter: Mike Heffner
Fix For: 1.0.11, 1.1.3

Attachments: 4462.txt

[jira] [Updated] (CASSANDRA-4462) upgradesstables strips active data from sstables

[
https://issues.apache.org/jira/browse/CASSANDRA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis updated CASSANDRA-4462:
--

Reviewer: jbellis
Affects Version/s: (was: 1.1.2)
1.0.4
Fix Version/s: 1.0.11
Assignee: Sylvain Lebresne

upgradesstables strips active data from sstables

Attachments: 4462.txt

[jira] [Updated] (CASSANDRA-4240) Only check the size of indexed column values when they are of type KEYS

2012-07-26 Thread T Jake Luciani (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-4240:
--

Assignee: Alex Liu

 Only check the size of indexed column values when they are of type KEYS
 ---

 Key: CASSANDRA-4240
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4240
 Project: Cassandra
  Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Joaquin Casares
Assignee: Alex Liu
Priority: Minor
  Labels: datastax_qa
 Fix For: 1.0.11

 Attachments: CASSANDRA-4240.patch, CASSANDRA-4240.patch1, 
 cassandra-1.0.8-CASSANDRA-4240-patch2.txt, 
 cassandra-1.0.8-CASSANDRA-4240-patch3.txt, 
 cassandra-1.0.8-CASSANDRA-4240-patch4.txt, 
 cassandra-1.0.8-CASSANDRA-4240-patch5.txt, cassandra-1.0.8-CASSANDRA-4240.txt


 https://github.com/apache/cassandra/blob/cassandra-1.0.8/src/java/org/apache/cassandra/thrift/ThriftValidation.java#L431
 That line states that: Indexed column values cannot be larger than 64K. But 
 in some cases we would want the column values to be able to be larger than 
 64k, specifically if the index_type is not of type KEYS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

git commit: cqlsh: fix problem describing pre-cql3 CFs that are case-sensitive Patch by paul cannon reviewed by brandonwilliams for CASSANDRA-4385

2012-07-26 Thread brandonwilliams

Updated Branches:
  refs/heads/cassandra-1.1 bb37a0f25 - 10372c20d


cqlsh: fix problem describing pre-cql3 CFs that are case-sensitive
Patch by paul cannon reviewed by brandonwilliams for CASSANDRA-4385


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10372c20
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10372c20
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10372c20

Branch: refs/heads/cassandra-1.1
Commit: 10372c20dfe5af0dc0ab7ed77370cd785c78e82e
Parents: bb37a0f
Author: Brandon Williams brandonwilli...@apache.org
Authored: Thu Jul 26 12:38:10 2012 -0500
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Thu Jul 26 12:38:10 2012 -0500

--
 bin/cqlsh |9 -
 1 files changed, 8 insertions(+), 1 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/10372c20/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index c67a818..c4b35fc 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -329,6 +329,9 @@ class NoKeyspaceError(Exception):
 class KeyspaceNotFound(Exception):
 pass
 
+class ColumnFamilyNotFound(Exception):
+pass
+
 class VersionNotSupported(Exception):
 pass
 
@@ -678,6 +681,8 @@ class Shell(cmd.Cmd):
 
 def fetchdict(self):
 row = self.cursor.fetchone()
+if row is None:
+return None
 desc = self.cursor.description
 return dict(zip([d[0] for d in desc], row))
 
@@ -705,7 +710,7 @@ class Shell(cmd.Cmd):
 for c in cf_defs:
 if c.name == cfname:
 return c
-raise KeyError(Unconfigured column family %r % (cfname,))
+raise ColumnFamilyNotFound(Unconfigured column family %r % (cfname,))
 
 def get_columnfamily_names(self, ksname=None):
 return [c.name for c in self.get_columnfamilies(ksname)]
@@ -800,6 +805,8 @@ class Shell(cmd.Cmd):
 where keyspace=:ks and columnfamily=:cf,
 {'ks': ksname, 'cf': cfname})
 layout = self.fetchdict()
+if layout is None:
+raise ColumnFamilyNotFound(Column family %r not found % cfname)
 self.cursor.execute(select * from system.schema_columns
 where keyspace=:ks and columnfamily=:cf,
 {'ks': ksname, 'cf': cfname})

[1/2] git commit: Merge branch 'cassandra-1.1' into trunk

2012-07-26 Thread brandonwilliams

Updated Branches:
  refs/heads/trunk 1507ce8cd - 94e413f5a


Merge branch 'cassandra-1.1' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/94e413f5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/94e413f5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/94e413f5

Branch: refs/heads/trunk
Commit: 94e413f5a127f374728a0b9ade931acc1d442c84
Parents: 1507ce8 10372c2
Author: Brandon Williams brandonwilli...@apache.org
Authored: Thu Jul 26 12:39:41 2012 -0500
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Thu Jul 26 12:39:41 2012 -0500

--
 bin/cqlsh |9 -
 1 files changed, 8 insertions(+), 1 deletions(-)
--

[2/2] git commit: cqlsh: fix problem describing pre-cql3 CFs that are case-sensitive Patch by paul cannon reviewed by brandonwilliams for CASSANDRA-4385

2012-07-26 Thread brandonwilliams

cqlsh: fix problem describing pre-cql3 CFs that are case-sensitive
Patch by paul cannon reviewed by brandonwilliams for CASSANDRA-4385


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10372c20
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10372c20
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10372c20

Branch: refs/heads/trunk
Commit: 10372c20dfe5af0dc0ab7ed77370cd785c78e82e
Parents: bb37a0f
Author: Brandon Williams brandonwilli...@apache.org
Authored: Thu Jul 26 12:38:10 2012 -0500
Committer: Brandon Williams brandonwilli...@apache.org
Committed: Thu Jul 26 12:38:10 2012 -0500

--
 bin/cqlsh |9 -
 1 files changed, 8 insertions(+), 1 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/10372c20/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index c67a818..c4b35fc 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -329,6 +329,9 @@ class NoKeyspaceError(Exception):
 class KeyspaceNotFound(Exception):
 pass
 
+class ColumnFamilyNotFound(Exception):
+pass
+
 class VersionNotSupported(Exception):
 pass
 
@@ -678,6 +681,8 @@ class Shell(cmd.Cmd):
 
 def fetchdict(self):
 row = self.cursor.fetchone()
+if row is None:
+return None
 desc = self.cursor.description
 return dict(zip([d[0] for d in desc], row))
 
@@ -705,7 +710,7 @@ class Shell(cmd.Cmd):
 for c in cf_defs:
 if c.name == cfname:
 return c
-raise KeyError(Unconfigured column family %r % (cfname,))
+raise ColumnFamilyNotFound(Unconfigured column family %r % (cfname,))
 
 def get_columnfamily_names(self, ksname=None):
 return [c.name for c in self.get_columnfamilies(ksname)]
@@ -800,6 +805,8 @@ class Shell(cmd.Cmd):
 where keyspace=:ks and columnfamily=:cf,
 {'ks': ksname, 'cf': cfname})
 layout = self.fetchdict()
+if layout is None:
+raise ColumnFamilyNotFound(Column family %r not found % cfname)
 self.cursor.execute(select * from system.schema_columns
 where keyspace=:ks and columnfamily=:cf,
 {'ks': ksname, 'cf': cfname})

git commit: Push the validation of secondary index values to the secondary index manager

2012-07-26 Thread jake

Updated Branches:
  refs/heads/cassandra-1.0 787e0e648 - bce2d6c35


Push the validation of secondary index values to the secondary index manager

Patch by Alex Liu, Reviewed by tjake for CASSANDRA-4240


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bce2d6c3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bce2d6c3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bce2d6c3

Branch: refs/heads/cassandra-1.0
Commit: bce2d6c35753929c62a557159f944ace82896869
Parents: 787e0e6
Author: T Jake Luciani jak...@gmail.com
Authored: Thu Jul 26 13:37:24 2012 -0400
Committer: T Jake Luciani jak...@gmail.com
Committed: Thu Jul 26 13:37:24 2012 -0400

--
 CHANGES.txt|2 +-
 .../db/index/PerColumnSecondaryIndex.java  |8 +
 .../cassandra/db/index/PerRowSecondaryIndex.java   |7 +
 .../apache/cassandra/db/index/SecondaryIndex.java  |3 +
 .../cassandra/db/index/SecondaryIndexManager.java  |7 +
 .../apache/cassandra/thrift/ThriftValidation.java  |   14 +-
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |  216 +++
 7 files changed, 249 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 66a7b9e..79278aa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -11,7 +11,7 @@
  * Set gc_grace on index CF to 0 (CASSANDRA-4314)
  * fix 1.0.x node join to mixed version cluster, other nodes = 1.1 
(CASSANDRA-4195)
  * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419)
-
+ * Push the validation of secondary index values to the SecondaryIndexManager 
(CASSANDRA-4240)
 
 1.0.10
  * fix maxTimestamp to include row tombstones (CASSANDRA-4116)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
index c137ead..a7bd3fb 100644
--- a/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
@@ -22,6 +22,8 @@ import java.nio.ByteBuffer;
 
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.IColumn;
+import org.apache.cassandra.thrift.Column;
+import org.apache.cassandra.utils.FBUtilities;
 
 /**
  * Base class for Secondary indexes that implement a unique index per column
@@ -60,4 +62,10 @@ public abstract class PerColumnSecondaryIndex extends 
SecondaryIndex
 {
 return getIndexName();   
 }
+
+@Override
+public boolean validate(Column column)
+{
+return column.value.remaining()  FBUtilities.MAX_UNSIGNED_SHORT;
+}
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
index f23980b..a14fead 100644
--- a/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
@@ -26,6 +26,7 @@ import java.util.SortedSet;
 import org.apache.cassandra.db.ColumnFamily;
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.IColumn;
+import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
 
 /**
@@ -70,4 +71,10 @@ public abstract class PerRowSecondaryIndex extends 
SecondaryIndex
 throw new RuntimeException(e);
 }
 }
+
+@Override
+public boolean validate(Column column)
+{
+return true;
+}
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
index 5006217..3e38f28 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
@@ -31,6 +31,7 @@ import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.index.keys.KeysIndex;
 import org.apache.cassandra.io.sstable.ReducingKeyIterator;
 import org.apache.cassandra.io.sstable.SSTableReader;
+import

git commit: remove poorly mocked test in favor of new test

2012-07-26 Thread jake

Updated Branches:
  refs/heads/cassandra-1.0 bce2d6c35 - 27cd2bfda


remove poorly mocked test in favor of new test


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/27cd2bfd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/27cd2bfd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/27cd2bfd

Branch: refs/heads/cassandra-1.0
Commit: 27cd2bfda07ac9dc2b336a45aadfb96a168a2d2b
Parents: bce2d6c
Author: T Jake Luciani jak...@gmail.com
Authored: Thu Jul 26 14:35:31 2012 -0400
Committer: T Jake Luciani jak...@gmail.com
Committed: Thu Jul 26 14:35:31 2012 -0400

--
 .../cassandra/thrift/ThriftValidationTest.java |   45 +--
 1 files changed, 1 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/27cd2bfd/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
--
diff --git a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java 
b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
index e550144..bcbbbf4 100644
--- a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
+++ b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
@@ -52,50 +52,7 @@ public class ThriftValidationTest extends CleanupHelper
 @Test
 public void testColumnValueSizeForIndexedColumn() throws 
ConfigurationException, InvalidRequestException
 {
-CfDef cfDef = Schema.instance.getCFMetaData(Keyspace1, 
Standard1).toThrift();
-ByteBuffer columnName = AsciiType.instance.fromString(indexed);
-
-// add an indexed column definition
-cfDef.addToColumn_metadata(new ColumnDef(columnName, 
UTF8Type.class.getCanonicalName())
- .setIndex_type(IndexType.KEYS)
- 
.setIndex_name(indexed_col));
-
-CFMetaData metaData = CFMetaData.fromThrift(cfDef);
-
-Column column = new Column(columnName)
-.setValue(new byte[FBUtilities.MAX_UNSIGNED_SHORT 
+ 1])
-.setTimestamp(System.currentTimeMillis());
-
-boolean gotException = false;
-
-try
-{
-// this run should throw an exception
-ThriftValidation.validateColumnData(metaData, column, false);
-}
-catch (InvalidRequestException e)
-{
-gotException = true;
-}
-
-assert gotException : expected InvalidRequestException but not 
received.;
-
-// change value to be less than unsigned short size
-column.setValue(new byte[12]);
-
-gotException = false; // reset flag
-
-try
-{
-// this run should run clean
-ThriftValidation.validateColumnData(metaData, column, false);
-}
-catch (InvalidRequestException e)
-{
-gotException = true;
-}
-
-assert !gotException : got unexpected InvalidRequestException;
+   //Replaced by SecondaryIndexColumnSizeTest
 }
 
 @Test

[jira] [Updated] (CASSANDRA-4009) Increase usage of Metrics and flesh out o.a.c.metrics

2012-07-26 Thread Yuki Morishita (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yuki Morishita updated CASSANDRA-4009:
--

Attachment: 4009.txt

Attaching patch for review.

* Removed EndpointMetrics because I felt those are not metrics. You can get
equivalent info from FailureDetectorMBean's AllEndpointStates.
* Some metrics now are Counters.
* Deprecated MBeans' attributes (nodetool still uses deprecated interface. I
want to do that in separate ticket).

on github: https://github.com/yukim/cassandra/tree/4009

Also updated doc: https://gist.github.com/3013075

Increase usage of Metrics and flesh out o.a.c.metrics
-

Key: CASSANDRA-4009
URL: https://issues.apache.org/jira/browse/CASSANDRA-4009
Project: Cassandra
Issue Type: Improvement
Reporter: Brandon Williams
Assignee: Yuki Morishita
Priority: Minor
Fix For: 1.2

Attachments: 4009.txt, 4009.txt

With CASSANDRA-3671 we have begun using the Metrics packages to expose stats
in a new JMX structure, intended to be more user-friendly (for example, you
don't need to know what a StorageProxy is or does.) This ticket serves as a
parent for subtasks to finish fleshing out the rest of the enhanced metrics.

[4/13] git commit: fix typo in SICST class name

fix typo in SICST class name


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1ad710b6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1ad710b6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1ad710b6

Branch: refs/heads/cassandra-1.1
Commit: 1ad710b6eada9b0b269c96154021a95a807f4bbb
Parents: bcc7f7b
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:12:24 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:12:24 2012 -0500

--
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |  215 ---
 .../cassandra/db/SecondaryIndexColumnSizeTest.java |  215 +++
 2 files changed, 215 insertions(+), 215 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1ad710b6/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java 
b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
deleted file mode 100644
index 1f6a0a1..000
--- a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
+++ /dev/null
@@ -1,215 +0,0 @@
-/*
-* Licensed to the Apache Software Foundation (ASF) under one
-* or more contributor license agreements.  See the NOTICE file
-* distributed with this work for additional information
-* regarding copyright ownership.  The ASF licenses this file
-* to you under the Apache License, Version 2.0 (the
-* License); you may not use this file except in compliance
-* with the License.  You may obtain a copy of the License at
-*
-*http://www.apache.org/licenses/LICENSE-2.0
-*
-* Unless required by applicable law or agreed to in writing,
-* software distributed under the License is distributed on an
-* AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-* KIND, either express or implied.  See the License for the
-* specific language governing permissions and limitations
-* under the License.
-*/
-package org.apache.cassandra.db;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.List;
-import java.util.Set;
-import java.util.SortedSet;
-
-import org.junit.Test;
-
-import org.apache.cassandra.config.ConfigurationException;
-import org.apache.cassandra.db.index.PerColumnSecondaryIndex;
-import org.apache.cassandra.db.index.PerRowSecondaryIndex;
-import org.apache.cassandra.db.index.SecondaryIndexSearcher;
-import org.apache.cassandra.thrift.Column;
-import org.apache.cassandra.utils.ByteBufferUtil;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-
-public class SecondaryIndexColumSizeTest
-{
-@Test
-public void test64kColumn()
-{
-Column column = new Column();
-column.name = ByteBufferUtil.bytes(test);
- 
-// a byte buffer more than 64k
-ByteBuffer buffer = ByteBuffer.allocate(1024 * 65);
-buffer.clear();
-
-//read more than 64k
-for (int i=0; i1024*64/4 + 1; i++)
-buffer.putInt(0);
-
-// for read
-buffer.flip(); 
-column.value = buffer;
-
-MockRowIndex mockRowIndex = new MockRowIndex();
-MockColumnIndex mockColumnIndex = new MockColumnIndex();
-
-assertTrue(mockRowIndex.validate(column));
-assertFalse(mockColumnIndex.validate(column));
-
-// test less than 64k value
-buffer.flip();
-buffer.clear();
-buffer.putInt(20);
-buffer.flip();
-
-assertTrue(mockRowIndex.validate(column));
-assertTrue(mockColumnIndex.validate(column));
-}
-
-private class MockRowIndex extends PerRowSecondaryIndex
-{
-@Override
-public void applyIndexUpdates(ByteBuffer rowKey, ColumnFamily cf, 
SortedSetByteBuffer mutatedIndexedColumns, ColumnFamily oldIndexedColumns) 
throws IOException 
-{
-}
-
-@Override
-public void init() 
-{   
-}
-
-@Override
-public void validateOptions() throws ConfigurationException 
-{
-}
-
-@Override
-public String getIndexName() 
-{
-return null;
-}
-
-@Override
-protected SecondaryIndexSearcher 
createSecondaryIndexSearcher(SetByteBuffer columns) 
-{
-return null;
-}
-
-@Override
-public void forceBlockingFlush() throws IOException 
-{
-}
-
-@Override
-public long getLiveSize() 
-{
-return 0;
-}
-
-@Override
-public ColumnFamilyStore getIndexCfs() 
-{
-return null;
-}
-
-

[2/13] git commit: merge from 1.0

merge from 1.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/070c1d8d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/070c1d8d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/070c1d8d

Branch: refs/heads/trunk
Commit: 070c1d8dd3c7b674034acfbc421555d658aabb60
Parents: 10372c2 1ad710b
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:16:39 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:16:39 2012 -0500

--
 CHANGES.txt|3 +
 .../db/index/PerColumnSecondaryIndex.java  |8 +
 .../cassandra/db/index/PerRowSecondaryIndex.java   |7 +
 .../apache/cassandra/db/index/SecondaryIndex.java  |3 +
 .../cassandra/db/index/SecondaryIndexManager.java  |   19 +-
 .../apache/cassandra/thrift/ThriftValidation.java  |   14 +-
 .../cassandra/db/SecondaryIndexColumnSizeTest.java |  215 +++
 .../cassandra/thrift/ThriftValidationTest.java |   52 
 8 files changed, 258 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/070c1d8d/CHANGES.txt
--
diff --cc CHANGES.txt
index 84db73d,79278aa..bcf56b7
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,154 -1,6 +1,155 @@@
 -1.0.11
 +1.1.3
 + * (JMX) rename getRangeKeySample to sampleKeyRange to avoid returning
 +   multi-MB results as an attribute (CASSANDRA-4452)
 + * flush based on data size, not throughput; overwritten columns no 
 +   longer artificially inflate liveRatio (CASSANDRA-4399)
 + * update default commitlog segment size to 32MB and total commitlog
 +   size to 32/1024 MB for 32/64 bit JVMs, respectively (CASSANDRA-4422)
 + * avoid using global partitioner to estimate ranges in index sstables
 +   (CASSANDRA-4403)
 + * restore pre-CASSANDRA-3862 approach to removing expired tombstones
 +   from row cache during compaction (CASSANDRA-4364)
 + * (stress) support for CQL prepared statements (CASSANDRA-3633)
 + * Correctly catch exception when Snappy cannot be loaded (CASSANDRA-4400)
 + * (cql3) Support ORDER BY when IN condition is given in WHERE clause 
(CASSANDRA-4327)
 + * (cql3) delete component_index column on DROP TABLE call (CASSANDRA-4420)
 + * change nanoTime() to currentTimeInMillis() in schema related code 
(CASSANDRA-4432)
 + * add a token generation tool (CASSANDRA-3709)
 + * Fix LCS bug with sstable containing only 1 row (CASSANDRA-4411)
 + * fix Can't Modify Index Name problem on CF update (CASSANDRA-4439)
 + * Fix assertion error in getOverlappingSSTables during repair 
(CASSANDRA-4456)
 + * fix nodetool's setcompactionthreshold command (CASSANDRA-4455)
 + * Ensure compacted files are never used, to avoid counter overcount 
(CASSANDRA-4436)
 +Merged from 1.0:
++ * Push the validation of secondary index values to the SecondaryIndexManager 
(CASSANDRA-4240)
   * allow dropping columns shadowed by not-yet-expired supercolumn or row
 tombstones in PrecompactedRow (CASSANDRA-4396)
 + * fix 1.0.x node join to mixed version cluster, other nodes = 1.1 
(CASSANDRA-4195)
 + * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419)
 + * Bootstraps that fail are detected upon restart and will retry safely 
without
 +   needing to delete existing data first (CASSANDRA-4427)
 + * (cqlsh) add a COPY TO command to copy a CF to a CSV file (CASSANDRA-4434)
 +
 +
 +1.1.2
 + * Fix cleanup not deleting index entries (CASSANDRA-4379)
 + * Use correct partitioner when saving + loading caches (CASSANDRA-4331)
 + * Check schema before trying to export sstable (CASSANDRA-2760)
 + * Raise a meaningful exception instead of NPE when PFS encounters
 +   an unconfigured node + no default (CASSANDRA-4349)
 + * fix bug in sstable blacklisting with LCS (CASSANDRA-4343)
 + * LCS no longer promotes tiny sstables out of L0 (CASSANDRA-4341)
 + * skip tombstones during hint replay (CASSANDRA-4320)
 + * fix NPE in compactionstats (CASSANDRA-4318)
 + * enforce 1m min keycache for auto (CASSANDRA-4306)
 + * Have DeletedColumn.isMFD always return true (CASSANDRA-4307)
 + * (cql3) exeption message for ORDER BY constraints said primary filter can be
 +an IN clause, which is misleading (CASSANDRA-4319)
 + * (cql3) Reject (not yet supported) creation of 2ndardy indexes on tables 
with
 +   composite primary keys (CASSANDRA-4328)
 + * Set JVM stack size to 160k for java 7 (CASSANDRA-4275)
 + * cqlsh: add COPY command to load data from CSV flat files (CASSANDRA-4012)
 + * CFMetaData.fromThrift to throw ConfigurationException upon error 
(CASSANDRA-4353)
 + * Use CF comparator to sort indexed columns in SecondaryIndexManager
 +   (CASSANDRA-4365)
 + * add strategy_options to the KSMetaData.toString() output (CASSANDRA-4248)
 +

[5/13] git commit: fix typo in SICST class name

fix typo in SICST class name


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1ad710b6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1ad710b6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1ad710b6

Branch: refs/heads/trunk
Commit: 1ad710b6eada9b0b269c96154021a95a807f4bbb
Parents: bcc7f7b
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:12:24 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:12:24 2012 -0500

--
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |  215 ---
 .../cassandra/db/SecondaryIndexColumnSizeTest.java |  215 +++
 2 files changed, 215 insertions(+), 215 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1ad710b6/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java 
b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
deleted file mode 100644
index 1f6a0a1..000
--- a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
+++ /dev/null
@@ -1,215 +0,0 @@
-/*
-* Licensed to the Apache Software Foundation (ASF) under one
-* or more contributor license agreements.  See the NOTICE file
-* distributed with this work for additional information
-* regarding copyright ownership.  The ASF licenses this file
-* to you under the Apache License, Version 2.0 (the
-* License); you may not use this file except in compliance
-* with the License.  You may obtain a copy of the License at
-*
-*http://www.apache.org/licenses/LICENSE-2.0
-*
-* Unless required by applicable law or agreed to in writing,
-* software distributed under the License is distributed on an
-* AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-* KIND, either express or implied.  See the License for the
-* specific language governing permissions and limitations
-* under the License.
-*/
-package org.apache.cassandra.db;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.List;
-import java.util.Set;
-import java.util.SortedSet;
-
-import org.junit.Test;
-
-import org.apache.cassandra.config.ConfigurationException;
-import org.apache.cassandra.db.index.PerColumnSecondaryIndex;
-import org.apache.cassandra.db.index.PerRowSecondaryIndex;
-import org.apache.cassandra.db.index.SecondaryIndexSearcher;
-import org.apache.cassandra.thrift.Column;
-import org.apache.cassandra.utils.ByteBufferUtil;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-
-public class SecondaryIndexColumSizeTest
-{
-@Test
-public void test64kColumn()
-{
-Column column = new Column();
-column.name = ByteBufferUtil.bytes(test);
- 
-// a byte buffer more than 64k
-ByteBuffer buffer = ByteBuffer.allocate(1024 * 65);
-buffer.clear();
-
-//read more than 64k
-for (int i=0; i1024*64/4 + 1; i++)
-buffer.putInt(0);
-
-// for read
-buffer.flip(); 
-column.value = buffer;
-
-MockRowIndex mockRowIndex = new MockRowIndex();
-MockColumnIndex mockColumnIndex = new MockColumnIndex();
-
-assertTrue(mockRowIndex.validate(column));
-assertFalse(mockColumnIndex.validate(column));
-
-// test less than 64k value
-buffer.flip();
-buffer.clear();
-buffer.putInt(20);
-buffer.flip();
-
-assertTrue(mockRowIndex.validate(column));
-assertTrue(mockColumnIndex.validate(column));
-}
-
-private class MockRowIndex extends PerRowSecondaryIndex
-{
-@Override
-public void applyIndexUpdates(ByteBuffer rowKey, ColumnFamily cf, 
SortedSetByteBuffer mutatedIndexedColumns, ColumnFamily oldIndexedColumns) 
throws IOException 
-{
-}
-
-@Override
-public void init() 
-{   
-}
-
-@Override
-public void validateOptions() throws ConfigurationException 
-{
-}
-
-@Override
-public String getIndexName() 
-{
-return null;
-}
-
-@Override
-protected SecondaryIndexSearcher 
createSecondaryIndexSearcher(SetByteBuffer columns) 
-{
-return null;
-}
-
-@Override
-public void forceBlockingFlush() throws IOException 
-{
-}
-
-@Override
-public long getLiveSize() 
-{
-return 0;
-}
-
-@Override
-public ColumnFamilyStore getIndexCfs() 
-{
-return null;
-}
-
-@Override
-

[12/13] git commit: Push the validation of secondary index values to the secondary index manager

Push the validation of secondary index values to the secondary index manager

Patch by Alex Liu, Reviewed by tjake for CASSANDRA-4240


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bce2d6c3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bce2d6c3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bce2d6c3

Branch: refs/heads/trunk
Commit: bce2d6c35753929c62a557159f944ace82896869
Parents: 787e0e6
Author: T Jake Luciani jak...@gmail.com
Authored: Thu Jul 26 13:37:24 2012 -0400
Committer: T Jake Luciani jak...@gmail.com
Committed: Thu Jul 26 13:37:24 2012 -0400

--
 CHANGES.txt|2 +-
 .../db/index/PerColumnSecondaryIndex.java  |8 +
 .../cassandra/db/index/PerRowSecondaryIndex.java   |7 +
 .../apache/cassandra/db/index/SecondaryIndex.java  |3 +
 .../cassandra/db/index/SecondaryIndexManager.java  |7 +
 .../apache/cassandra/thrift/ThriftValidation.java  |   14 +-
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |  216 +++
 7 files changed, 249 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 66a7b9e..79278aa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -11,7 +11,7 @@
  * Set gc_grace on index CF to 0 (CASSANDRA-4314)
  * fix 1.0.x node join to mixed version cluster, other nodes = 1.1 
(CASSANDRA-4195)
  * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419)
-
+ * Push the validation of secondary index values to the SecondaryIndexManager 
(CASSANDRA-4240)
 
 1.0.10
  * fix maxTimestamp to include row tombstones (CASSANDRA-4116)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
index c137ead..a7bd3fb 100644
--- a/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
@@ -22,6 +22,8 @@ import java.nio.ByteBuffer;
 
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.IColumn;
+import org.apache.cassandra.thrift.Column;
+import org.apache.cassandra.utils.FBUtilities;
 
 /**
  * Base class for Secondary indexes that implement a unique index per column
@@ -60,4 +62,10 @@ public abstract class PerColumnSecondaryIndex extends 
SecondaryIndex
 {
 return getIndexName();   
 }
+
+@Override
+public boolean validate(Column column)
+{
+return column.value.remaining()  FBUtilities.MAX_UNSIGNED_SHORT;
+}
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
index f23980b..a14fead 100644
--- a/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
@@ -26,6 +26,7 @@ import java.util.SortedSet;
 import org.apache.cassandra.db.ColumnFamily;
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.IColumn;
+import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
 
 /**
@@ -70,4 +71,10 @@ public abstract class PerRowSecondaryIndex extends 
SecondaryIndex
 throw new RuntimeException(e);
 }
 }
+
+@Override
+public boolean validate(Column column)
+{
+return true;
+}
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
index 5006217..3e38f28 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
@@ -31,6 +31,7 @@ import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.index.keys.KeysIndex;
 import org.apache.cassandra.io.sstable.ReducingKeyIterator;
 import org.apache.cassandra.io.sstable.SSTableReader;
+import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import

[1/13] git commit: merge from 1.1

Updated Branches:
  refs/heads/cassandra-1.0 27cd2bfda - 1ad710b6e
  refs/heads/cassandra-1.1 10372c20d - 070c1d8dd
  refs/heads/trunk 94e413f5a - 80baefddc


merge from 1.1


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/80baefdd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/80baefdd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/80baefdd

Branch: refs/heads/trunk
Commit: 80baefddc5749a15edc21b6ea9fa84bdfb879883
Parents: 94e413f 070c1d8
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:17:21 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:17:21 2012 -0500

--
 CHANGES.txt|3 +
 .../db/index/PerColumnSecondaryIndex.java  |8 +
 .../cassandra/db/index/PerRowSecondaryIndex.java   |7 +
 .../apache/cassandra/db/index/SecondaryIndex.java  |3 +
 .../cassandra/db/index/SecondaryIndexManager.java  |   19 +-
 .../apache/cassandra/thrift/ThriftValidation.java  |   14 +-
 .../cassandra/db/SecondaryIndexColumnSizeTest.java |  215 +++
 .../cassandra/thrift/ThriftValidationTest.java |   58 +
 8 files changed, 259 insertions(+), 68 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/80baefdd/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/80baefdd/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/80baefdd/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/80baefdd/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/80baefdd/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/80baefdd/src/java/org/apache/cassandra/thrift/ThriftValidation.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/80baefdd/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
--
diff --cc test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
index d57f9c1,af0e668..04f0d0f
--- a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
+++ b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
@@@ -24,19 -24,15 +24,12 @@@ package org.apache.cassandra.thrift
  import org.junit.Test;
  
  import org.apache.cassandra.SchemaLoader;
--import org.apache.cassandra.config.CFMetaData;
--import org.apache.cassandra.config.ConfigurationException;
--import org.apache.cassandra.config.ColumnDefinition;
--import org.apache.cassandra.config.KSMetaData;
--import org.apache.cassandra.config.Schema;
++import org.apache.cassandra.config.*;
  import org.apache.cassandra.db.marshal.AsciiType;
  import org.apache.cassandra.db.marshal.UTF8Type;
  import org.apache.cassandra.locator.LocalStrategy;
  import org.apache.cassandra.locator.NetworkTopologyStrategy;
 +import org.apache.cassandra.utils.ByteBufferUtil;
- import org.apache.cassandra.utils.FBUtilities;
  
  public class ThriftValidationTest extends SchemaLoader
  {

[3/13] git commit: merge from 1.0

merge from 1.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/070c1d8d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/070c1d8d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/070c1d8d

Branch: refs/heads/cassandra-1.1
Commit: 070c1d8dd3c7b674034acfbc421555d658aabb60
Parents: 10372c2 1ad710b
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:16:39 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:16:39 2012 -0500

--
 CHANGES.txt|3 +
 .../db/index/PerColumnSecondaryIndex.java  |8 +
 .../cassandra/db/index/PerRowSecondaryIndex.java   |7 +
 .../apache/cassandra/db/index/SecondaryIndex.java  |3 +
 .../cassandra/db/index/SecondaryIndexManager.java  |   19 +-
 .../apache/cassandra/thrift/ThriftValidation.java  |   14 +-
 .../cassandra/db/SecondaryIndexColumnSizeTest.java |  215 +++
 .../cassandra/thrift/ThriftValidationTest.java |   52 
 8 files changed, 258 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/070c1d8d/CHANGES.txt
--
diff --cc CHANGES.txt
index 84db73d,79278aa..bcf56b7
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,154 -1,6 +1,155 @@@
 -1.0.11
 +1.1.3
 + * (JMX) rename getRangeKeySample to sampleKeyRange to avoid returning
 +   multi-MB results as an attribute (CASSANDRA-4452)
 + * flush based on data size, not throughput; overwritten columns no 
 +   longer artificially inflate liveRatio (CASSANDRA-4399)
 + * update default commitlog segment size to 32MB and total commitlog
 +   size to 32/1024 MB for 32/64 bit JVMs, respectively (CASSANDRA-4422)
 + * avoid using global partitioner to estimate ranges in index sstables
 +   (CASSANDRA-4403)
 + * restore pre-CASSANDRA-3862 approach to removing expired tombstones
 +   from row cache during compaction (CASSANDRA-4364)
 + * (stress) support for CQL prepared statements (CASSANDRA-3633)
 + * Correctly catch exception when Snappy cannot be loaded (CASSANDRA-4400)
 + * (cql3) Support ORDER BY when IN condition is given in WHERE clause 
(CASSANDRA-4327)
 + * (cql3) delete component_index column on DROP TABLE call (CASSANDRA-4420)
 + * change nanoTime() to currentTimeInMillis() in schema related code 
(CASSANDRA-4432)
 + * add a token generation tool (CASSANDRA-3709)
 + * Fix LCS bug with sstable containing only 1 row (CASSANDRA-4411)
 + * fix Can't Modify Index Name problem on CF update (CASSANDRA-4439)
 + * Fix assertion error in getOverlappingSSTables during repair 
(CASSANDRA-4456)
 + * fix nodetool's setcompactionthreshold command (CASSANDRA-4455)
 + * Ensure compacted files are never used, to avoid counter overcount 
(CASSANDRA-4436)
 +Merged from 1.0:
++ * Push the validation of secondary index values to the SecondaryIndexManager 
(CASSANDRA-4240)
   * allow dropping columns shadowed by not-yet-expired supercolumn or row
 tombstones in PrecompactedRow (CASSANDRA-4396)
 + * fix 1.0.x node join to mixed version cluster, other nodes = 1.1 
(CASSANDRA-4195)
 + * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419)
 + * Bootstraps that fail are detected upon restart and will retry safely 
without
 +   needing to delete existing data first (CASSANDRA-4427)
 + * (cqlsh) add a COPY TO command to copy a CF to a CSV file (CASSANDRA-4434)
 +
 +
 +1.1.2
 + * Fix cleanup not deleting index entries (CASSANDRA-4379)
 + * Use correct partitioner when saving + loading caches (CASSANDRA-4331)
 + * Check schema before trying to export sstable (CASSANDRA-2760)
 + * Raise a meaningful exception instead of NPE when PFS encounters
 +   an unconfigured node + no default (CASSANDRA-4349)
 + * fix bug in sstable blacklisting with LCS (CASSANDRA-4343)
 + * LCS no longer promotes tiny sstables out of L0 (CASSANDRA-4341)
 + * skip tombstones during hint replay (CASSANDRA-4320)
 + * fix NPE in compactionstats (CASSANDRA-4318)
 + * enforce 1m min keycache for auto (CASSANDRA-4306)
 + * Have DeletedColumn.isMFD always return true (CASSANDRA-4307)
 + * (cql3) exeption message for ORDER BY constraints said primary filter can be
 +an IN clause, which is misleading (CASSANDRA-4319)
 + * (cql3) Reject (not yet supported) creation of 2ndardy indexes on tables 
with
 +   composite primary keys (CASSANDRA-4328)
 + * Set JVM stack size to 160k for java 7 (CASSANDRA-4275)
 + * cqlsh: add COPY command to load data from CSV flat files (CASSANDRA-4012)
 + * CFMetaData.fromThrift to throw ConfigurationException upon error 
(CASSANDRA-4353)
 + * Use CF comparator to sort indexed columns in SecondaryIndexManager
 +   (CASSANDRA-4365)
 + * add strategy_options to the KSMetaData.toString() output

[8/13] git commit: cleanup

cleanup


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bcc7f7b0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bcc7f7b0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bcc7f7b0

Branch: refs/heads/cassandra-1.1
Commit: bcc7f7b0fbd0eabfc1ae985009cce7fca985b613
Parents: 27cd2bf
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:12:02 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:12:02 2012 -0500

--
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |   29 +++
 .../cassandra/thrift/ThriftValidationTest.java |9 -
 2 files changed, 14 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bcc7f7b0/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java 
b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
index bf35c30..1f6a0a1 100644
--- a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
+++ b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
@@ -17,8 +17,6 @@
 * under the License.
 */
 package org.apache.cassandra.db;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
@@ -26,17 +24,20 @@ import java.util.List;
 import java.util.Set;
 import java.util.SortedSet;
 
+import org.junit.Test;
+
 import org.apache.cassandra.config.ConfigurationException;
 import org.apache.cassandra.db.index.PerColumnSecondaryIndex;
 import org.apache.cassandra.db.index.PerRowSecondaryIndex;
 import org.apache.cassandra.db.index.SecondaryIndexSearcher;
 import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
-import org.junit.Test;
 
-public class SecondaryIndexColumSizeTest 
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class SecondaryIndexColumSizeTest
 {
-
 @Test
 public void test64kColumn()
 {
@@ -55,11 +56,11 @@ public class SecondaryIndexColumSizeTest
 buffer.flip(); 
 column.value = buffer;
 
-SolrIndex solrIndex = new SolrIndex();
-ColumnIndex perColumnIndex = new ColumnIndex();
+MockRowIndex mockRowIndex = new MockRowIndex();
+MockColumnIndex mockColumnIndex = new MockColumnIndex();
 
-assertTrue(solrIndex.validate(column));
-assertFalse(perColumnIndex.validate(column));
+assertTrue(mockRowIndex.validate(column));
+assertFalse(mockColumnIndex.validate(column));
 
 // test less than 64k value
 buffer.flip();
@@ -67,13 +68,12 @@ public class SecondaryIndexColumSizeTest
 buffer.putInt(20);
 buffer.flip();
 
-assertTrue(solrIndex.validate(column));
-assertTrue(perColumnIndex.validate(column));
+assertTrue(mockRowIndex.validate(column));
+assertTrue(mockColumnIndex.validate(column));
 }
 
-public class SolrIndex extends PerRowSecondaryIndex
+private class MockRowIndex extends PerRowSecondaryIndex
 {
-
 @Override
 public void applyIndexUpdates(ByteBuffer rowKey, ColumnFamily cf, 
SortedSetByteBuffer mutatedIndexedColumns, ColumnFamily oldIndexedColumns) 
throws IOException 
 {
@@ -141,9 +141,8 @@ public class SecondaryIndexColumSizeTest
 }
 
 
-public class ColumnIndex extends PerColumnSecondaryIndex
+private class MockColumnIndex extends PerColumnSecondaryIndex
 {
-
 @Override
 public void init() 
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bcc7f7b0/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
--
diff --git a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java 
b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
index bcbbbf4..c025037 100644
--- a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
+++ b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
@@ -21,8 +21,6 @@ package org.apache.cassandra.thrift;
  */
 
 
-import java.nio.ByteBuffer;
-
 import org.junit.Test;
 
 import org.apache.cassandra.CleanupHelper;
@@ -33,7 +31,6 @@ import org.apache.cassandra.db.marshal.AsciiType;
 import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.locator.LocalStrategy;
 import org.apache.cassandra.locator.NetworkTopologyStrategy;
-import org.apache.cassandra.utils.FBUtilities;
 
 public class ThriftValidationTest extends CleanupHelper
 {
@@ -50,12

[7/13] git commit: cleanup

cleanup


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bcc7f7b0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bcc7f7b0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bcc7f7b0

Branch: refs/heads/trunk
Commit: bcc7f7b0fbd0eabfc1ae985009cce7fca985b613
Parents: 27cd2bf
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:12:02 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:12:02 2012 -0500

--
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |   29 +++
 .../cassandra/thrift/ThriftValidationTest.java |9 -
 2 files changed, 14 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bcc7f7b0/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java 
b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
index bf35c30..1f6a0a1 100644
--- a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
+++ b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
@@ -17,8 +17,6 @@
 * under the License.
 */
 package org.apache.cassandra.db;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
@@ -26,17 +24,20 @@ import java.util.List;
 import java.util.Set;
 import java.util.SortedSet;
 
+import org.junit.Test;
+
 import org.apache.cassandra.config.ConfigurationException;
 import org.apache.cassandra.db.index.PerColumnSecondaryIndex;
 import org.apache.cassandra.db.index.PerRowSecondaryIndex;
 import org.apache.cassandra.db.index.SecondaryIndexSearcher;
 import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
-import org.junit.Test;
 
-public class SecondaryIndexColumSizeTest 
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class SecondaryIndexColumSizeTest
 {
-
 @Test
 public void test64kColumn()
 {
@@ -55,11 +56,11 @@ public class SecondaryIndexColumSizeTest
 buffer.flip(); 
 column.value = buffer;
 
-SolrIndex solrIndex = new SolrIndex();
-ColumnIndex perColumnIndex = new ColumnIndex();
+MockRowIndex mockRowIndex = new MockRowIndex();
+MockColumnIndex mockColumnIndex = new MockColumnIndex();
 
-assertTrue(solrIndex.validate(column));
-assertFalse(perColumnIndex.validate(column));
+assertTrue(mockRowIndex.validate(column));
+assertFalse(mockColumnIndex.validate(column));
 
 // test less than 64k value
 buffer.flip();
@@ -67,13 +68,12 @@ public class SecondaryIndexColumSizeTest
 buffer.putInt(20);
 buffer.flip();
 
-assertTrue(solrIndex.validate(column));
-assertTrue(perColumnIndex.validate(column));
+assertTrue(mockRowIndex.validate(column));
+assertTrue(mockColumnIndex.validate(column));
 }
 
-public class SolrIndex extends PerRowSecondaryIndex
+private class MockRowIndex extends PerRowSecondaryIndex
 {
-
 @Override
 public void applyIndexUpdates(ByteBuffer rowKey, ColumnFamily cf, 
SortedSetByteBuffer mutatedIndexedColumns, ColumnFamily oldIndexedColumns) 
throws IOException 
 {
@@ -141,9 +141,8 @@ public class SecondaryIndexColumSizeTest
 }
 
 
-public class ColumnIndex extends PerColumnSecondaryIndex
+private class MockColumnIndex extends PerColumnSecondaryIndex
 {
-
 @Override
 public void init() 
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bcc7f7b0/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
--
diff --git a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java 
b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
index bcbbbf4..c025037 100644
--- a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
+++ b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
@@ -21,8 +21,6 @@ package org.apache.cassandra.thrift;
  */
 
 
-import java.nio.ByteBuffer;
-
 import org.junit.Test;
 
 import org.apache.cassandra.CleanupHelper;
@@ -33,7 +31,6 @@ import org.apache.cassandra.db.marshal.AsciiType;
 import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.locator.LocalStrategy;
 import org.apache.cassandra.locator.NetworkTopologyStrategy;
-import org.apache.cassandra.utils.FBUtilities;
 
 public class ThriftValidationTest extends CleanupHelper
 {
@@ -50,12 +47,6 @@

[13/13] git commit: Push the validation of secondary index values to the secondary index manager

Push the validation of secondary index values to the secondary index manager

Patch by Alex Liu, Reviewed by tjake for CASSANDRA-4240


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bce2d6c3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bce2d6c3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bce2d6c3

Branch: refs/heads/cassandra-1.1
Commit: bce2d6c35753929c62a557159f944ace82896869
Parents: 787e0e6
Author: T Jake Luciani jak...@gmail.com
Authored: Thu Jul 26 13:37:24 2012 -0400
Committer: T Jake Luciani jak...@gmail.com
Committed: Thu Jul 26 13:37:24 2012 -0400

--
 CHANGES.txt|2 +-
 .../db/index/PerColumnSecondaryIndex.java  |8 +
 .../cassandra/db/index/PerRowSecondaryIndex.java   |7 +
 .../apache/cassandra/db/index/SecondaryIndex.java  |3 +
 .../cassandra/db/index/SecondaryIndexManager.java  |7 +
 .../apache/cassandra/thrift/ThriftValidation.java  |   14 +-
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |  216 +++
 7 files changed, 249 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 66a7b9e..79278aa 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -11,7 +11,7 @@
  * Set gc_grace on index CF to 0 (CASSANDRA-4314)
  * fix 1.0.x node join to mixed version cluster, other nodes = 1.1 
(CASSANDRA-4195)
  * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419)
-
+ * Push the validation of secondary index values to the SecondaryIndexManager 
(CASSANDRA-4240)
 
 1.0.10
  * fix maxTimestamp to include row tombstones (CASSANDRA-4116)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
index c137ead..a7bd3fb 100644
--- a/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/PerColumnSecondaryIndex.java
@@ -22,6 +22,8 @@ import java.nio.ByteBuffer;
 
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.IColumn;
+import org.apache.cassandra.thrift.Column;
+import org.apache.cassandra.utils.FBUtilities;
 
 /**
  * Base class for Secondary indexes that implement a unique index per column
@@ -60,4 +62,10 @@ public abstract class PerColumnSecondaryIndex extends 
SecondaryIndex
 {
 return getIndexName();   
 }
+
+@Override
+public boolean validate(Column column)
+{
+return column.value.remaining()  FBUtilities.MAX_UNSIGNED_SHORT;
+}
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
index f23980b..a14fead 100644
--- a/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/PerRowSecondaryIndex.java
@@ -26,6 +26,7 @@ import java.util.SortedSet;
 import org.apache.cassandra.db.ColumnFamily;
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.IColumn;
+import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
 
 /**
@@ -70,4 +71,10 @@ public abstract class PerRowSecondaryIndex extends 
SecondaryIndex
 throw new RuntimeException(e);
 }
 }
+
+@Override
+public boolean validate(Column column)
+{
+return true;
+}
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bce2d6c3/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
index 5006217..3e38f28 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
@@ -31,6 +31,7 @@ import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.index.keys.KeysIndex;
 import org.apache.cassandra.io.sstable.ReducingKeyIterator;
 import org.apache.cassandra.io.sstable.SSTableReader;
+import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
 import

[9/13] git commit: cleanup

cleanup


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bcc7f7b0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bcc7f7b0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bcc7f7b0

Branch: refs/heads/cassandra-1.0
Commit: bcc7f7b0fbd0eabfc1ae985009cce7fca985b613
Parents: 27cd2bf
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:12:02 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:12:02 2012 -0500

--
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |   29 +++
 .../cassandra/thrift/ThriftValidationTest.java |9 -
 2 files changed, 14 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bcc7f7b0/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java 
b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
index bf35c30..1f6a0a1 100644
--- a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
+++ b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
@@ -17,8 +17,6 @@
 * under the License.
 */
 package org.apache.cassandra.db;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
@@ -26,17 +24,20 @@ import java.util.List;
 import java.util.Set;
 import java.util.SortedSet;
 
+import org.junit.Test;
+
 import org.apache.cassandra.config.ConfigurationException;
 import org.apache.cassandra.db.index.PerColumnSecondaryIndex;
 import org.apache.cassandra.db.index.PerRowSecondaryIndex;
 import org.apache.cassandra.db.index.SecondaryIndexSearcher;
 import org.apache.cassandra.thrift.Column;
 import org.apache.cassandra.utils.ByteBufferUtil;
-import org.junit.Test;
 
-public class SecondaryIndexColumSizeTest 
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class SecondaryIndexColumSizeTest
 {
-
 @Test
 public void test64kColumn()
 {
@@ -55,11 +56,11 @@ public class SecondaryIndexColumSizeTest
 buffer.flip(); 
 column.value = buffer;
 
-SolrIndex solrIndex = new SolrIndex();
-ColumnIndex perColumnIndex = new ColumnIndex();
+MockRowIndex mockRowIndex = new MockRowIndex();
+MockColumnIndex mockColumnIndex = new MockColumnIndex();
 
-assertTrue(solrIndex.validate(column));
-assertFalse(perColumnIndex.validate(column));
+assertTrue(mockRowIndex.validate(column));
+assertFalse(mockColumnIndex.validate(column));
 
 // test less than 64k value
 buffer.flip();
@@ -67,13 +68,12 @@ public class SecondaryIndexColumSizeTest
 buffer.putInt(20);
 buffer.flip();
 
-assertTrue(solrIndex.validate(column));
-assertTrue(perColumnIndex.validate(column));
+assertTrue(mockRowIndex.validate(column));
+assertTrue(mockColumnIndex.validate(column));
 }
 
-public class SolrIndex extends PerRowSecondaryIndex
+private class MockRowIndex extends PerRowSecondaryIndex
 {
-
 @Override
 public void applyIndexUpdates(ByteBuffer rowKey, ColumnFamily cf, 
SortedSetByteBuffer mutatedIndexedColumns, ColumnFamily oldIndexedColumns) 
throws IOException 
 {
@@ -141,9 +141,8 @@ public class SecondaryIndexColumSizeTest
 }
 
 
-public class ColumnIndex extends PerColumnSecondaryIndex
+private class MockColumnIndex extends PerColumnSecondaryIndex
 {
-
 @Override
 public void init() 
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bcc7f7b0/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
--
diff --git a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java 
b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
index bcbbbf4..c025037 100644
--- a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
+++ b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
@@ -21,8 +21,6 @@ package org.apache.cassandra.thrift;
  */
 
 
-import java.nio.ByteBuffer;
-
 import org.junit.Test;
 
 import org.apache.cassandra.CleanupHelper;
@@ -33,7 +31,6 @@ import org.apache.cassandra.db.marshal.AsciiType;
 import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.locator.LocalStrategy;
 import org.apache.cassandra.locator.NetworkTopologyStrategy;
-import org.apache.cassandra.utils.FBUtilities;
 
 public class ThriftValidationTest extends CleanupHelper
 {
@@ -50,12

[10/13] git commit: remove poorly mocked test in favor of new test

remove poorly mocked test in favor of new test


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/27cd2bfd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/27cd2bfd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/27cd2bfd

Branch: refs/heads/trunk
Commit: 27cd2bfda07ac9dc2b336a45aadfb96a168a2d2b
Parents: bce2d6c
Author: T Jake Luciani jak...@gmail.com
Authored: Thu Jul 26 14:35:31 2012 -0400
Committer: T Jake Luciani jak...@gmail.com
Committed: Thu Jul 26 14:35:31 2012 -0400

--
 .../cassandra/thrift/ThriftValidationTest.java |   45 +--
 1 files changed, 1 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/27cd2bfd/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
--
diff --git a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java 
b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
index e550144..bcbbbf4 100644
--- a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
+++ b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
@@ -52,50 +52,7 @@ public class ThriftValidationTest extends CleanupHelper
 @Test
 public void testColumnValueSizeForIndexedColumn() throws 
ConfigurationException, InvalidRequestException
 {
-CfDef cfDef = Schema.instance.getCFMetaData(Keyspace1, 
Standard1).toThrift();
-ByteBuffer columnName = AsciiType.instance.fromString(indexed);
-
-// add an indexed column definition
-cfDef.addToColumn_metadata(new ColumnDef(columnName, 
UTF8Type.class.getCanonicalName())
- .setIndex_type(IndexType.KEYS)
- 
.setIndex_name(indexed_col));
-
-CFMetaData metaData = CFMetaData.fromThrift(cfDef);
-
-Column column = new Column(columnName)
-.setValue(new byte[FBUtilities.MAX_UNSIGNED_SHORT 
+ 1])
-.setTimestamp(System.currentTimeMillis());
-
-boolean gotException = false;
-
-try
-{
-// this run should throw an exception
-ThriftValidation.validateColumnData(metaData, column, false);
-}
-catch (InvalidRequestException e)
-{
-gotException = true;
-}
-
-assert gotException : expected InvalidRequestException but not 
received.;
-
-// change value to be less than unsigned short size
-column.setValue(new byte[12]);
-
-gotException = false; // reset flag
-
-try
-{
-// this run should run clean
-ThriftValidation.validateColumnData(metaData, column, false);
-}
-catch (InvalidRequestException e)
-{
-gotException = true;
-}
-
-assert !gotException : got unexpected InvalidRequestException;
+   //Replaced by SecondaryIndexColumnSizeTest
 }
 
 @Test

[11/13] git commit: remove poorly mocked test in favor of new test

remove poorly mocked test in favor of new test


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/27cd2bfd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/27cd2bfd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/27cd2bfd

Branch: refs/heads/cassandra-1.1
Commit: 27cd2bfda07ac9dc2b336a45aadfb96a168a2d2b
Parents: bce2d6c
Author: T Jake Luciani jak...@gmail.com
Authored: Thu Jul 26 14:35:31 2012 -0400
Committer: T Jake Luciani jak...@gmail.com
Committed: Thu Jul 26 14:35:31 2012 -0400

--
 .../cassandra/thrift/ThriftValidationTest.java |   45 +--
 1 files changed, 1 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/27cd2bfd/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
--
diff --git a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java 
b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
index e550144..bcbbbf4 100644
--- a/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
+++ b/test/unit/org/apache/cassandra/thrift/ThriftValidationTest.java
@@ -52,50 +52,7 @@ public class ThriftValidationTest extends CleanupHelper
 @Test
 public void testColumnValueSizeForIndexedColumn() throws 
ConfigurationException, InvalidRequestException
 {
-CfDef cfDef = Schema.instance.getCFMetaData(Keyspace1, 
Standard1).toThrift();
-ByteBuffer columnName = AsciiType.instance.fromString(indexed);
-
-// add an indexed column definition
-cfDef.addToColumn_metadata(new ColumnDef(columnName, 
UTF8Type.class.getCanonicalName())
- .setIndex_type(IndexType.KEYS)
- 
.setIndex_name(indexed_col));
-
-CFMetaData metaData = CFMetaData.fromThrift(cfDef);
-
-Column column = new Column(columnName)
-.setValue(new byte[FBUtilities.MAX_UNSIGNED_SHORT 
+ 1])
-.setTimestamp(System.currentTimeMillis());
-
-boolean gotException = false;
-
-try
-{
-// this run should throw an exception
-ThriftValidation.validateColumnData(metaData, column, false);
-}
-catch (InvalidRequestException e)
-{
-gotException = true;
-}
-
-assert gotException : expected InvalidRequestException but not 
received.;
-
-// change value to be less than unsigned short size
-column.setValue(new byte[12]);
-
-gotException = false; // reset flag
-
-try
-{
-// this run should run clean
-ThriftValidation.validateColumnData(metaData, column, false);
-}
-catch (InvalidRequestException e)
-{
-gotException = true;
-}
-
-assert !gotException : got unexpected InvalidRequestException;
+   //Replaced by SecondaryIndexColumnSizeTest
 }
 
 @Test

[6/13] git commit: fix typo in SICST class name

fix typo in SICST class name


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1ad710b6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1ad710b6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1ad710b6

Branch: refs/heads/cassandra-1.0
Commit: 1ad710b6eada9b0b269c96154021a95a807f4bbb
Parents: bcc7f7b
Author: Jonathan Ellis jbel...@apache.org
Authored: Thu Jul 26 17:12:24 2012 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Thu Jul 26 17:12:24 2012 -0500

--
 .../cassandra/db/SecondaryIndexColumSizeTest.java  |  215 ---
 .../cassandra/db/SecondaryIndexColumnSizeTest.java |  215 +++
 2 files changed, 215 insertions(+), 215 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1ad710b6/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java 
b/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
deleted file mode 100644
index 1f6a0a1..000
--- a/test/unit/org/apache/cassandra/db/SecondaryIndexColumSizeTest.java
+++ /dev/null
@@ -1,215 +0,0 @@
-/*
-* Licensed to the Apache Software Foundation (ASF) under one
-* or more contributor license agreements.  See the NOTICE file
-* distributed with this work for additional information
-* regarding copyright ownership.  The ASF licenses this file
-* to you under the Apache License, Version 2.0 (the
-* License); you may not use this file except in compliance
-* with the License.  You may obtain a copy of the License at
-*
-*http://www.apache.org/licenses/LICENSE-2.0
-*
-* Unless required by applicable law or agreed to in writing,
-* software distributed under the License is distributed on an
-* AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-* KIND, either express or implied.  See the License for the
-* specific language governing permissions and limitations
-* under the License.
-*/
-package org.apache.cassandra.db;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.List;
-import java.util.Set;
-import java.util.SortedSet;
-
-import org.junit.Test;
-
-import org.apache.cassandra.config.ConfigurationException;
-import org.apache.cassandra.db.index.PerColumnSecondaryIndex;
-import org.apache.cassandra.db.index.PerRowSecondaryIndex;
-import org.apache.cassandra.db.index.SecondaryIndexSearcher;
-import org.apache.cassandra.thrift.Column;
-import org.apache.cassandra.utils.ByteBufferUtil;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-
-public class SecondaryIndexColumSizeTest
-{
-@Test
-public void test64kColumn()
-{
-Column column = new Column();
-column.name = ByteBufferUtil.bytes(test);
- 
-// a byte buffer more than 64k
-ByteBuffer buffer = ByteBuffer.allocate(1024 * 65);
-buffer.clear();
-
-//read more than 64k
-for (int i=0; i1024*64/4 + 1; i++)
-buffer.putInt(0);
-
-// for read
-buffer.flip(); 
-column.value = buffer;
-
-MockRowIndex mockRowIndex = new MockRowIndex();
-MockColumnIndex mockColumnIndex = new MockColumnIndex();
-
-assertTrue(mockRowIndex.validate(column));
-assertFalse(mockColumnIndex.validate(column));
-
-// test less than 64k value
-buffer.flip();
-buffer.clear();
-buffer.putInt(20);
-buffer.flip();
-
-assertTrue(mockRowIndex.validate(column));
-assertTrue(mockColumnIndex.validate(column));
-}
-
-private class MockRowIndex extends PerRowSecondaryIndex
-{
-@Override
-public void applyIndexUpdates(ByteBuffer rowKey, ColumnFamily cf, 
SortedSetByteBuffer mutatedIndexedColumns, ColumnFamily oldIndexedColumns) 
throws IOException 
-{
-}
-
-@Override
-public void init() 
-{   
-}
-
-@Override
-public void validateOptions() throws ConfigurationException 
-{
-}
-
-@Override
-public String getIndexName() 
-{
-return null;
-}
-
-@Override
-protected SecondaryIndexSearcher 
createSecondaryIndexSearcher(SetByteBuffer columns) 
-{
-return null;
-}
-
-@Override
-public void forceBlockingFlush() throws IOException 
-{
-}
-
-@Override
-public long getLiveSize() 
-{
-return 0;
-}
-
-@Override
-public ColumnFamilyStore getIndexCfs() 
-{
-return null;
-}
-
-

[jira] [Updated] (CASSANDRA-4443) shuffle utility for vnodes


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-4443:
--

Assignee: Eric Evans

Eric has some notes up at http://wiki.apache.org/cassandra/VirtualNodes/Balance 
on implementation but a wiki is a lousy discussion medium so I'll post here:

shuffle should operate on the entire cluster, at least by default.  Shuffling 
node at a time means that for each node i for i in 0..N-1 (where N is the 
cluster size), i/N of the ranges shuffled will, on average, have been shuffled 
at least once already.  So it's substantially less efficient than shuffling 
once, then assigning the vnodes out in one cluster-wide pass.

 shuffle utility for vnodes
 --

 Key: CASSANDRA-4443
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4443
 Project: Cassandra
  Issue Type: Sub-task
  Components: Core
Affects Versions: 1.2
Reporter: Brandon Williams
Assignee: Eric Evans
 Fix For: 1.2


 As mentioned on CASSANDRA-4127, for upgrades we need a 'shuffle' command to 
 split up the contiguous ranges.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4049) Add generic way of adding SSTable components required custom compaction strategy


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423513#comment-13423513
 ] 

Jonathan Ellis commented on CASSANDRA-4049:
---

Sorry for the delay, would you mind rebasing to current trunk?

 Add generic way of adding SSTable components required custom compaction 
 strategy
 

 Key: CASSANDRA-4049
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4049
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Piotr Kołaczkowski
Assignee: Piotr Kołaczkowski
Priority: Minor
  Labels: compaction
 Fix For: 1.1.4

 Attachments: pluggable_custom_components.patch


 CFS compaction strategy coming up in the next DSE release needs to store some 
 important information in Tombstones.db and RemovedKeys.db files, one per 
 sstable. However, currently Cassandra issues warnings when these files are 
 found in the data directory. Additionally, when switched to 
 SizeTieredCompactionStrategy, the files are left in the data directory after 
 compaction.
 The attached patch adds new components to the Component class so Cassandra 
 knows about those files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Cassandra Wiki] Update of VirtualNodes/Balance by EricEvans

2012-07-26 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The VirtualNodes/Balance page has been changed by EricEvans:
http://wiki.apache.org/cassandra/VirtualNodes/Balance?action=diffrev1=4rev2=5

  Anchor(implementation)
  == Implementation (Draft) ==
  === Considerations ===
- In the most basic sense, ''balanced'' means that each node has 1/n of the 
token-space, so adjusting ownership for [[#heterogeneous_nodes|heterogeneous 
nodes]] is implicitly about ''unbalancing''.  This is important because, if for 
example, you reduced ownership of a node to say (1/n)*.8, you expect that 
imbalance to persist, and not be balanced-away by operations on other nodes.
+  * In the most basic sense, ''balanced'' means that each node has 1/n of the 
token-space, so adjusting ownership for [[#heterogeneous_nodes|heterogeneous 
nodes]] is implicitly about ''unbalancing''.  This is important because, if for 
example, you reduced ownership of a node to say (1/n)*.8, you expect that 
imbalance to persist, and not be balanced-away by operations on other nodes.
  
- ''Note: This will likely require storing state, in the form of an offset, on 
each node.''
+  * Shuffling node at a time means that for each node i for i in 0..N-1 
(where N is the cluster size), i/N of the ranges shuffled will, on average, 
have been shuffled at least once already. So it's substantially less efficient 
than shuffling once, then assigning the vnodes out in one cluster-wide pass. 
-- ''Jonathan 
Ellis''FootNote([[https://issues.apache.org/jira/browse/CASSANDRA-4443?focusedCommentId=13423505page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13423505|CASSANDRA-4443#comment-13423505]])
  
  === Nodes / Cluster ===
  The most straightforward method of effecting ownership is a token move (i.e. 
relocating a range from one node to another).  Exposing this with JMX would 
allow implementing all of the required operations client-side.
  
  === User Interface ===
  
- The `balance` sub-command balances the node it is ran against, by default a 
targeted ownership of `1/n`.  The sub-command takes an optional offset in the 
rangeFootNote(Does this range make sense?) of `+100` to `-100`, which 
results in a targeted ownership of `(1/n)*(offset/100)`.
+ TODO
  
- ''Note: ranges copied from/to other nodes must be selected in such a way as 
to respect their offsets.''
- 
- {{{
- $ nodetool balance [+/-offset]
- }}}
- 
- The ``shuffle` sub-command randomly exchanges contiguous ranges on the node 
it ran against, with other nodes in the cluster.
- 
- {{{
- $ nodetool shuffle
- }}}
- 
- The `trim` sub-command assigns an offset in the rangeFootNote(Does this 
range make sense?) of `+100` to `-100`, and copies randomly selected ranges 
onto, or off of, the node it is ran against to achieve the requested ownership 
(`(1/n)*(offset/100)`).
- 
- {{{
- $ nodetool trim +/- offset
- }}}
-

[jira] [Commented] (CASSANDRA-4443) shuffle utility for vnodes

2012-07-26 Thread Eric Evans (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423519#comment-13423519
]

Eric Evans commented on CASSANDRA-4443:
---

bq. Eric has some notes up at
http://wiki.apache.org/cassandra/VirtualNodes/Balance on implementation but a
wiki is a lousy discussion medium so I'll post here:

It is, but so is Jira. :)

I'm not trying to start a discussion on the wiki, I just wanted to get down the
general requirements and brain-storm a bit so that any subsequent discussion
(if needed) might be more productive.

bq. shuffle should operate on the entire cluster, at least by default.
Shuffling node at a time means that for each node i for i in 0..N-1 (where N is
the cluster size), i/N of the ranges shuffled will, on average, have been
shuffled at least once already. So it's substantially less efficient than
shuffling once, then assigning the vnodes out in one cluster-wide pass.

Yeah, this is exactly the sort of thing I was trying to capture (and I've added
it).

shuffle utility for vnodes
--

Key: CASSANDRA-4443
URL: https://issues.apache.org/jira/browse/CASSANDRA-4443
Project: Cassandra
Issue Type: Sub-task
Components: Core
Affects Versions: 1.2
Reporter: Brandon Williams
Assignee: Eric Evans
Fix For: 1.2

As mentioned on CASSANDRA-4127, for upgrades we need a 'shuffle' command to
split up the contiguous ranges.

[jira] [Updated] (CASSANDRA-2858) make request dropping more accurate


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2858:
--

Description: Based on the discussion in CASSANDRA-2819, we can make the 
bookkeeping for request times more accurate.  (was: Based on the discussion in 
https://issues.apache.org/jira/browse/CASSANDRA-2819, we can make the 
bookkeeping for request times more accurate.)

 make request dropping more accurate
 ---

 Key: CASSANDRA-2858
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2858
 Project: Cassandra
  Issue Type: Improvement
Reporter: Ryan King
Assignee: Melvin Wang
Priority: Minor
 Fix For: 1.2


 Based on the discussion in CASSANDRA-2819, we can make the bookkeeping for 
 request times more accurate.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4443) shuffle utility for vnodes


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423550#comment-13423550
 ] 

Jonathan Ellis commented on CASSANDRA-4443:
---

Thanks!

 shuffle utility for vnodes
 --

 Key: CASSANDRA-4443
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4443
 Project: Cassandra
  Issue Type: Sub-task
  Components: Core
Affects Versions: 1.2
Reporter: Brandon Williams
Assignee: Eric Evans
 Fix For: 1.2


 As mentioned on CASSANDRA-4127, for upgrades we need a 'shuffle' command to 
 split up the contiguous ranges.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

git commit: fix compile