[jira] [Created] (CASSANDRA-2774) one way to make counter delete work better
one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Reporter: Yang Yang current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049649#comment-13049649 ] Yang Yang commented on CASSANDRA-2774: -- see original jira https://issues.apache.org/jira/browse/CASSANDRA-2101 one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Reporter: Yang Yang current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yang updated CASSANDRA-2774: - Attachment: counter_delete.diff note that the main logic is rather simple, and concentrated in CounterColumn.reconcile() but most of the coding was done around the issue of setting up a lastDeleteTimestamp() for a completely new incoming CounterUpdateColumn, since it does not any history yet. the code in this part uses some rather messy changes, and should definitely use a better route, but so far it's only for demonstration of the idea. people more familiar with the code path can suggest a better way. one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049651#comment-13049651 ] Yang Yang commented on CASSANDRA-2774: -- note that the original idea is to use the epoch number, which is a pure traditional logical clock, but I found that we already have the var of timestampOfLastDelete(), this has exactly the same effect, so I just re-used the timestampOfLastDelete() for the purpose of epoch number. one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049650#comment-13049650 ] Yang Yang edited comment on CASSANDRA-2774 at 6/15/11 6:42 AM: --- note that the main logic is rather simple, and concentrated in CounterColumn.reconcile() but most of the coding was done around the issue of setting up a lastDeleteTimestamp() for a completely new incoming CounterUpdateColumn, since it does not any history yet. the code in this part uses some rather messy changes, and should definitely use a better route, but so far it's only for demonstration of the idea. people more familiar with the code path can suggest a better way. here is how it works: when a counterUpdateColumn comes in, we put it in memtable, which goes through the reconcile process. new code obtains the state of those columns listed in the mutation, AFTER the mutation is applied. we check whether any columns are completely new, i.e. they do not have a matching one in memtable, so that their timestampOfLastDelete() is still UNDECIDED. then for these columns, we do a read, and find any of their existing columns in SStables, and assign the correct timestampOfLastDelete() to them. we do not do the read in sstable at first because most counter adds will already have one matching column in memtable, so now we only incur the extra sstable reading cost when we start a new counter in memtable. reconcile rule: 1) newer epoch (timestampOfLastDelete() ) wins 2) UNDECIDED inherits the timestmapOfLastDelete() of the competitor (UNDECIDED can be seen ONLY when a new column comes into memtable) 3) if timestampOfLastDelete() is same, non-delete wins over delete 4) at last, use standard merging between counterColumns was (Author: yangyangyyy): note that the main logic is rather simple, and concentrated in CounterColumn.reconcile() but most of the coding was done around the issue of setting up a lastDeleteTimestamp() for a completely new incoming CounterUpdateColumn, since it does not any history yet. the code in this part uses some rather messy changes, and should definitely use a better route, but so far it's only for demonstration of the idea. people more familiar with the code path can suggest a better way. one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049654#comment-13049654 ] Yang Yang commented on CASSANDRA-2774: -- the patch is against standard 0.8.0-src source one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[
https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049688#comment-13049688
]
Sylvain Lebresne commented on CASSANDRA-2774:
-
Consider 2 nodes A, B and C with RF=2 and a given counter c whose replica set
is {B, C}.
Consider a single client issuing the following operations (in order) while
connected to node A:
# client increment c by +2 at CL.ONE
# client delete c at CL.ONE
# client increment c by +3 at CL.ONE
# client reads c at CL.ALL
The *only* valid answer the client should ever get on its last read is 3. Any
other value is a break of the consistency level contract and not something we
can expect people to be happy with. Any other answer means that deletes are
broken (and this *is* the problem with the actual implementation).
However, because the write are made at CL.ONE in the example above, at the time
the read is issued, the only thing we know for sure is that each write has been
received by one node, but not necessarily the same each time. Depending on the
actual timing and on which node happens to be the one acknowledging each
writes, when the read reaches the nodes you can have a lot of different
situations including:
* A and B both have received the 3 writes in the right order, they will all
return 3, the 'right' answer.
* A received the deletion (the two increments are still on the wire yet to be
received) and B received the other two increments (the delete is still on the
wire yet to be received). A will return the tombstone, B will return 5. You can
assign all epoch number you want, there is no way you can return 3 to the
client. It will be either 5 or 0.
So the same query will result in different answers depending on the internal
timing of events, and will sometimes return an answer that is a break of the
contract. Removes of counters are broken and the only safe way to use them is
for permanent removal with no following inserts. This patch doesn't fix it.
Btw, it's not too hard to come up with the same kind of example using only
QUORUM reads and writes (but you'll need one more replica and a few more steps).
one way to make counter delete work better
--
Key: CASSANDRA-2774
URL: https://issues.apache.org/jira/browse/CASSANDRA-2774
Project: Cassandra
Issue Type: New Feature
Affects Versions: 0.8.0
Reporter: Yang Yang
Attachments: counter_delete.diff
current Counter does not work with delete, because different merging order of
sstables would produces different result, for example:
add 1
delete
add 2
if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2
if merging is: 1--3, (1,3)--2, the result will be 3.
the issue is that delete now can not separate out previous adds and adds
later than the delete. supposedly a delete is to create a completely new
incarnation of the counter, or a new lifetime, or epoch. the new approach
utilizes the concept of epoch number, so that each delete bumps up the
epoch number. since each write is replicated (replicate on write is almost
always enabled in practice, if this is a concern, we could further force ROW
in case of delete ), so the epoch number is global to a replica set
changes are attached, existing tests pass fine, some tests are modified since
the semantic is changed a bit. some cql tests do not pass in the original
0.8.0 source, that's not the fault of this change.
see details at
http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E
the goal of this is to make delete work ( at least with consistent behavior,
yes in case of long network partition, the behavior is not ideal, but it's
consistent with the definition of logical clock), so that we could have
expiring Counters
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1135981 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/db/
Author: slebresne Date: Wed Jun 15 09:46:10 2011 New Revision: 1135981 URL: http://svn.apache.org/viewvc?rev=1135981view=rev Log: Merge from 0.8 Modified: cassandra/trunk/ (props changed) cassandra/trunk/CHANGES.txt cassandra/trunk/contrib/ (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/trunk/src/java/org/apache/cassandra/db/DataTracker.java Propchange: cassandra/trunk/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 09:46:10 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7:1026516-1133874 /cassandra/branches/cassandra-0.7.0:1053690-1055654 -/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135399,1135426,1135441 +/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441 /cassandra/branches/cassandra-0.8.0:1125021-1130369 /cassandra/branches/cassandra-0.8.1:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1135981r1=1135980r2=1135981view=diff == --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Wed Jun 15 09:46:10 2011 @@ -58,6 +58,8 @@ supercolumn tombstone during replica resolution (CASSANDRA-2590) * support sstable2json against snapshot sstables (CASSANDRA-2386) * remove active-pull schema requests (CASSANDRA-2715) + * avoid marking entire list of sstables as actively being compacted + in multithreaded compaction (CASSANDRA-2765) * seek back after deserializing a row to update cache with (CASSANDRA-2752) * avoid skipping rows in scrub for counter column family (CASSANDRA-2759) * fix ConcurrentModificationException in repair when dealing with 0.7 node Propchange: cassandra/trunk/contrib/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 09:46:10 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009 /cassandra/branches/cassandra-0.7/contrib:1026516-1133874 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654 -/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135399,1135426,1135441 +/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 09:46:10 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1133874 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 -/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135399,1135426,1135441 +/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441 /cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369 /cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 09:46:10 2011 @@ -1,7 +1,7 @@
[jira] [Created] (CASSANDRA-2775) ant build-test is failing with 0.8.0
ant build-test is failing with 0.8.0 Key: CASSANDRA-2775 URL: https://issues.apache.org/jira/browse/CASSANDRA-2775 Project: Cassandra Issue Type: Bug Components: Tests Affects Versions: 0.8.0 Reporter: Vivek Mishra Assignee: Vivek Mishra Priority: Minor Steps to reproduce: 1) ant clean build 2) ant build-test It is failing for : build-test: [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build.xml:975: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set [javac] Compiling 124 source files to C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build\test\classes [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] CassandraPreparedStatement stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] CassandraPreparedStatement stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac]^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:365: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:377: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 4 errors -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2775) ant build-test is failing with 0.8.0
[
https://issues.apache.org/jira/browse/CASSANDRA-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vivek Mishra updated CASSANDRA-2775:
Attachment: build.xml
Changed build.xml to modify cassandra.classpath for
pathelement location=${build.classes.cql} /
Made it working.
attached build.xml for review.
ant build-test is failing with 0.8.0
Key: CASSANDRA-2775
URL: https://issues.apache.org/jira/browse/CASSANDRA-2775
Project: Cassandra
Issue Type: Bug
Components: Tests
Affects Versions: 0.8.0
Reporter: Vivek Mishra
Assignee: Vivek Mishra
Priority: Minor
Attachments: build.xml
Steps to reproduce:
1) ant clean build
2) ant build-test
It is failing for :
build-test:
[javac]
C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build.xml:975:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set
[javac] Compiling 124 source files to
C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build\test\classes
[javac]
C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357:
cannot find symbol
[javac] symbol : class CassandraPreparedStatement
[javac] location: class
org.apache.cassandra.cql.jdbc.PreparedStatementTest
[javac] CassandraPreparedStatement stmt =
(CassandraPreparedStatement)con.prepareStatement(q);
[javac] ^
[javac]
C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357:
cannot find symbol
[javac] symbol : class CassandraPreparedStatement
[javac] location: class
org.apache.cassandra.cql.jdbc.PreparedStatementTest
[javac] CassandraPreparedStatement stmt =
(CassandraPreparedStatement)con.prepareStatement(q);
[javac]^
[javac]
C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:365:
cannot find symbol
[javac] symbol : class CassandraPreparedStatement
[javac] location: class
org.apache.cassandra.cql.jdbc.PreparedStatementTest
[javac] stmt =
(CassandraPreparedStatement)con.prepareStatement(q);
[javac] ^
[javac]
C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:377:
cannot find symbol
[javac] symbol : class CassandraPreparedStatement
[javac] location: class
org.apache.cassandra.cql.jdbc.PreparedStatementTest
[javac] stmt =
(CassandraPreparedStatement)con.prepareStatement(q);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2775) ant build-test is failing with 0.8.0
[ https://issues.apache.org/jira/browse/CASSANDRA-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049702#comment-13049702 ] Vivek Mishra commented on CASSANDRA-2775: - In case of not to modify cassandra.classpath should be create a different classpath for build-test? Build.xml attached with changes. ant build-test is failing with 0.8.0 Key: CASSANDRA-2775 URL: https://issues.apache.org/jira/browse/CASSANDRA-2775 Project: Cassandra Issue Type: Bug Components: Tests Affects Versions: 0.8.0 Reporter: Vivek Mishra Assignee: Vivek Mishra Priority: Minor Attachments: build.xml Steps to reproduce: 1) ant clean build 2) ant build-test It is failing for : build-test: [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build.xml:975: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set [javac] Compiling 124 source files to C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build\test\classes [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] CassandraPreparedStatement stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] CassandraPreparedStatement stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac]^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:365: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:377: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 4 errors -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2614) create Column and CounterColumn in the same column family
[ https://issues.apache.org/jira/browse/CASSANDRA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049714#comment-13049714 ] Chris Burroughs commented on CASSANDRA-2614: That's CQL in 0.8.1 for this ticket and a new one for Thrift api in 1.0? create Column and CounterColumn in the same column family - Key: CASSANDRA-2614 URL: https://issues.apache.org/jira/browse/CASSANDRA-2614 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Dave Rav Assignee: Sylvain Lebresne Priority: Minor Fix For: 0.8.1 create Column and CounterColumn in the same column family -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.
[
https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049724#comment-13049724
]
Mck SembWever commented on CASSANDRA-2388:
--
bq. [snip] One possibility is to use the ip octets like the
RackInferringSnitch.
In our usecase we have three nodes:{noformat}152.90.241.22=DC1:RAC1 #node1
152.90.241.23=DC2:RAC1 #node2
152.90.241.24=DC1:RAC1 #node3{noformat}
When we make the connection through the other (node2) endpoint taking the
rack inferring approach 152.90. will say it's in DC2. (again) this is the
wrong DC and will return itself as a valid endpoint
Step (3) seems to me to be too specific to be included here.
If i take it out we get this code:{noformat}public String[]
sort_endpoints_by_proximity(String endpoint, String[] endpoints, boolean
restrictToSameDC)
throws TException, InvalidRequestException
{
try
{
ListString results = new ArrayListString();
InetAddress address = InetAddress.getByName(endpoint);
boolean endpointValid = null !=
Gossiper.instance.getEndpointStateForEndpoint(address);
String datacenter = DatabaseDescriptor
.getEndpointSnitch().getDatacenter(endpointValid ? address
: FBUtilities.getLocalAddress());
ListInetAddress addresses = new ArrayListInetAddress();
for(String ep : endpoints)
{
addresses.add(InetAddress.getByName(endpoint));
}
DatabaseDescriptor.getEndpointSnitch().sortByProximity(address,
addresses);
for(InetAddress ep : addresses)
{
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(ep);
if(FailureDetector.instance.isAlive(ep) (!restrictToSameDC
|| datacenter.equals(dc)))
{
results.add(ep.getHostName());
}
}
return results.toArray(new String[results.size()]);
}
catch (UnknownHostException e)
{
throw new InvalidRequestException(e.getMessage());
}
}{noformat}
I'm happy with this (except that
{{Gossiper.instance.getEndpointStateForEndpoint(address)}} is only my guess on
how to tell if an endpoint is valid as such).
ColumnFamilyRecordReader fails for a given split because a host is down, even
if records could reasonably be read from other replica.
-
Key: CASSANDRA-2388
URL: https://issues.apache.org/jira/browse/CASSANDRA-2388
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Reporter: Eldon Stegall
Assignee: Mck SembWever
Labels: hadoop, inputformat
Fix For: 0.8.1
Attachments: 0002_On_TException_try_next_split.patch,
CASSANDRA-2388.patch, CASSANDRA-2388.patch
ColumnFamilyRecordReader only tries the first location for a given split. We
should try multiple locations for a given split.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.
[
https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049724#comment-13049724
]
Mck SembWever edited comment on CASSANDRA-2388 at 6/15/11 11:23 AM:
bq. [snip] One possibility is to use the ip octets like the
RackInferringSnitch.
In our usecase we have three nodes defined via
PropertyFileSnitch:{noformat}152.90.241.22=DC1:RAC1 #node1
152.90.241.23=DC2:RAC1 #node2
152.90.241.24=DC1:RAC1 #node3{noformat}
When we make the connection through the other (node2) endpoint taking the
rack inferring approach 152.90. will say it's in DC2. (again) this is the
wrong DC and will return itself as a valid endpoint
Step (3) seems to me to be too specific to be included here.
If i take it out we get this code:{noformat}public String[]
sort_endpoints_by_proximity(String endpoint, String[] endpoints, boolean
restrictToSameDC)
throws TException, InvalidRequestException
{
try
{
ListString results = new ArrayListString();
InetAddress address = InetAddress.getByName(endpoint);
boolean endpointValid = null !=
Gossiper.instance.getEndpointStateForEndpoint(address);
String datacenter = DatabaseDescriptor
.getEndpointSnitch().getDatacenter(endpointValid ? address
: FBUtilities.getLocalAddress());
ListInetAddress addresses = new ArrayListInetAddress();
for(String ep : endpoints)
{
addresses.add(InetAddress.getByName(endpoint));
}
DatabaseDescriptor.getEndpointSnitch().sortByProximity(address,
addresses);
for(InetAddress ep : addresses)
{
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(ep);
if(FailureDetector.instance.isAlive(ep) (!restrictToSameDC
|| datacenter.equals(dc)))
{
results.add(ep.getHostName());
}
}
return results.toArray(new String[results.size()]);
}
catch (UnknownHostException e)
{
throw new InvalidRequestException(e.getMessage());
}
}{noformat}
I'm happy with this (except that
{{Gossiper.instance.getEndpointStateForEndpoint(address)}} is only my guess on
how to tell if an endpoint is valid as such).
was (Author: michaelsembwever):
bq. [snip] One possibility is to use the ip octets like the
RackInferringSnitch.
In our usecase we have three nodes:{noformat}152.90.241.22=DC1:RAC1 #node1
152.90.241.23=DC2:RAC1 #node2
152.90.241.24=DC1:RAC1 #node3{noformat}
When we make the connection through the other (node2) endpoint taking the
rack inferring approach 152.90. will say it's in DC2. (again) this is the
wrong DC and will return itself as a valid endpoint
Step (3) seems to me to be too specific to be included here.
If i take it out we get this code:{noformat}public String[]
sort_endpoints_by_proximity(String endpoint, String[] endpoints, boolean
restrictToSameDC)
throws TException, InvalidRequestException
{
try
{
ListString results = new ArrayListString();
InetAddress address = InetAddress.getByName(endpoint);
boolean endpointValid = null !=
Gossiper.instance.getEndpointStateForEndpoint(address);
String datacenter = DatabaseDescriptor
.getEndpointSnitch().getDatacenter(endpointValid ? address
: FBUtilities.getLocalAddress());
ListInetAddress addresses = new ArrayListInetAddress();
for(String ep : endpoints)
{
addresses.add(InetAddress.getByName(endpoint));
}
DatabaseDescriptor.getEndpointSnitch().sortByProximity(address,
addresses);
for(InetAddress ep : addresses)
{
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(ep);
if(FailureDetector.instance.isAlive(ep) (!restrictToSameDC
|| datacenter.equals(dc)))
{
results.add(ep.getHostName());
}
}
return results.toArray(new String[results.size()]);
}
catch (UnknownHostException e)
{
throw new InvalidRequestException(e.getMessage());
}
}{noformat}
I'm happy with this (except that
{{Gossiper.instance.getEndpointStateForEndpoint(address)}} is only my guess on
how to tell if an endpoint is valid as such).
ColumnFamilyRecordReader fails for a given split because a host is down, even
if records could reasonably be read from other replica.
-
Key: CASSANDRA-2388
[jira] [Updated] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2769: Attachment: 0002-Only-compact-what-has-been-succesfully-marked-as-com.patch 0001-Do-compact-only-smallerSSTables.patch 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch Alright, there is a bunch of problems, one of which affects 0.8 and trunk and could cause this stackTrace. The others are due to CASSANDRA-1610 and thus only affect trunk (but one of those can also result in the attached stackTrace). The problem affecting 0.8 and trunk is related to a left over line in doCleanup() that is wrongly unmarking a sstable from the compacting set before having removed it from the active set of sstables. Thus another compaction could start compacting this sstable and we'll end up marking the file as compacted twice (and we would have duplicated the sstable, which is a problem for counters). Patch 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch removes it and is against 0.8. Trunk has a few problems of its own: * If disk space is not sufficient to compact all sstables, it computes the smallestSSTables set that fits, but doesn't use it. Attached first patch (0001-Do-compact-only-smallerSSTables.patch) fixes that. * The CompactionTask logic wrongly decorrelates the set of sstables that are successfully marked from the ones it did compact. That is, it grabs a list of sstables it wants to compact, then call markCompacting on them, but does not check if all of them are successfully marked and compact the original list instead. In effect, a task will recompact sstables that are already being compacted by other task and the given file will be compacted twice (or more) and marked compacted multiple times. Attached patch (0002-Only-compact-what-has-been-succesfully-marked-as-com.patch) fixes this by changing the sstables set of a given CompactionTask to whatever has been successfully marked only. Since the marking involves updating the task, I've move the logic to AbstractCompactionTask where it seems to make more sense to me. * For some reason, the markCompacting added for CompactionTasks was refusing to mark (and compact) anything if the set of sstable was bigger that MaxCompactionThreshold. This means that as soon as the number of sstables (of same size) in the column family would exceed the threshold, no compaction would be started. This is not the expected behavior. The second patch also fixes this by reusing the original markCompacting that handles this correctly. Cannot Create Duplicate Compaction Marker - Key: CASSANDRA-2769 URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.0 Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Fix For: 0.8.1, 1.0 Attachments: 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch, 0001-Do-compact-only-smallerSSTables.patch, 0002-Only-compact-what-has-been-succesfully-marked-as-com.patch Concurrent compaction can trigger the following exception when two threads compact the same sstable. DataTracker attempts to prevent this but apparently not successfully. java.io.IOError: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see:
svn commit: r1136006 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/utils/MerkleTree.java
Author: slebresne Date: Wed Jun 15 11:55:28 2011 New Revision: 1136006 URL: http://svn.apache.org/viewvc?rev=1136006view=rev Log: Avoids infinite loop when initializing a merkle tree patch by slebresne; reviewed by stuhood for CASSANDRA-2758 Modified: cassandra/branches/cassandra-0.8/CHANGES.txt cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/utils/MerkleTree.java Modified: cassandra/branches/cassandra-0.8/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1136006r1=1136005r2=1136006view=diff == --- cassandra/branches/cassandra-0.8/CHANGES.txt (original) +++ cassandra/branches/cassandra-0.8/CHANGES.txt Wed Jun 15 11:55:28 2011 @@ -56,6 +56,7 @@ * fix ConcurrentModificationException in repair when dealing with 0.7 node (CASSANDRA-2767) * use threadsafe collections for StreamInSession (CASSANDRA-2766) + * avoid infinite loop when creating merkle tree (CASSANDRA-2758) 0.8.0-final Modified: cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/utils/MerkleTree.java URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/utils/MerkleTree.java?rev=1136006r1=1136005r2=1136006view=diff == --- cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/utils/MerkleTree.java (original) +++ cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/utils/MerkleTree.java Wed Jun 15 11:55:28 2011 @@ -178,8 +178,11 @@ public class MerkleTree implements Seria return new Leaf(); Token midpoint = partitioner.midpoint(left, right); -Hashable lchild = midpoint.equals(left) ? new Leaf() : initHelper(left, midpoint, inc(depth), max); -Hashable rchild = midpoint.equals(right) ? new Leaf() : initHelper(midpoint, right, inc(depth), max); +if (midpoint.equals(left) || midpoint.equals(right)) +return new Leaf(); + +Hashable lchild = initHelper(left, midpoint, inc(depth), max); +Hashable rchild = initHelper(midpoint, right, inc(depth), max); return new Inner(midpoint, lchild, rchild); }
svn commit: r1136007 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/utils/
Author: slebresne Date: Wed Jun 15 11:57:44 2011 New Revision: 1136007 URL: http://svn.apache.org/viewvc?rev=1136007view=rev Log: Merge from 0.8 Modified: cassandra/trunk/ (props changed) cassandra/trunk/CHANGES.txt cassandra/trunk/contrib/ (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/trunk/src/java/org/apache/cassandra/utils/MerkleTree.java Propchange: cassandra/trunk/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 11:57:44 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7:1026516-1133874 /cassandra/branches/cassandra-0.7.0:1053690-1055654 -/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441 +/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441,1136006 /cassandra/branches/cassandra-0.8.0:1125021-1130369 /cassandra/branches/cassandra-0.8.1:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1136007r1=1136006r2=1136007view=diff == --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Wed Jun 15 11:57:44 2011 @@ -64,6 +64,7 @@ * avoid skipping rows in scrub for counter column family (CASSANDRA-2759) * fix ConcurrentModificationException in repair when dealing with 0.7 node (CASSANDRA-2767) + * avoid infinite loop when creating merkle tree (CASSANDRA-2758) 0.8.0-final Propchange: cassandra/trunk/contrib/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 11:57:44 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009 /cassandra/branches/cassandra-0.7/contrib:1026516-1133874 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654 -/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441 +/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441,1136006 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 11:57:44 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1133874 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 -/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441 +/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135288,1135399,1135426,1135441,1136006 /cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369 /cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Wed Jun 15 11:57:44 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
svn commit: r1136008 - in /cassandra/trunk: CHANGES.txt src/java/org/apache/cassandra/dht/AbstractBounds.java test/unit/org/apache/cassandra/dht/AbstractBoundsTest.java
Author: slebresne
Date: Wed Jun 15 12:01:21 2011
New Revision: 1136008
URL: http://svn.apache.org/viewvc?rev=1136008view=rev
Log:
Make AbstractBounds.normalize de-overlapp overlapping ranges
patch by slebresne; reviewed by stuhood for CASSANDRA-2641
Added:
cassandra/trunk/test/unit/org/apache/cassandra/dht/AbstractBoundsTest.java
Modified:
cassandra/trunk/CHANGES.txt
cassandra/trunk/src/java/org/apache/cassandra/dht/AbstractBounds.java
Modified: cassandra/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1136008r1=1136007r2=1136008view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Wed Jun 15 12:01:21 2011
@@ -2,6 +2,7 @@
* removed binarymemtable (CASSANDRA-2692)
* add commitlog_total_space_in_mb to prevent fragmented logs (CASSANDRA-2427)
* removed commitlog_rotation_threshold_in_mb configuration (CASSANDRA-2771)
+ * make AbstractBounds.normalize de-overlapp overlapping ranges
(CASSANDRA-2641)
0.8.1
Modified: cassandra/trunk/src/java/org/apache/cassandra/dht/AbstractBounds.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/dht/AbstractBounds.java?rev=1136008r1=1136007r2=1136008view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/dht/AbstractBounds.java
(original)
+++ cassandra/trunk/src/java/org/apache/cassandra/dht/AbstractBounds.java Wed
Jun 15 12:01:21 2011
@@ -89,8 +89,8 @@ public abstract class AbstractBounds imp
public abstract ListAbstractBounds unwrap();
/**
- * @return A copy of the given list of non-intersecting bounds with all
bounds unwrapped, sorted by bound.left.
- * This method does not allow overlapping ranges as input.
+ * @return A copy of the given list of with all bounds unwrapped, sorted
by bound.left and with overlapping bounds merged.
+ * This method does not allow allow to mix Range and Bound in the input
list.
*/
public static ListAbstractBounds normalize(Collection? extends
AbstractBounds bounds)
{
@@ -107,6 +107,61 @@ public abstract class AbstractBounds imp
return b1.left.compareTo(b2.left);
}
});
+
+// deoverlap
+return deoverlap(output);
+}
+
+/**
+ * Given a list of unwrapped bounds sorted by left token, return a list a
equivalent
+ * list of bounds but with no overlapping bounds.
+ */
+private static ListAbstractBounds deoverlap(ListAbstractBounds bounds)
+{
+if (bounds.isEmpty())
+return bounds;
+
+ListAbstractBounds output = new ArrayListAbstractBounds();
+
+IteratorAbstractBounds iter = bounds.iterator();
+AbstractBounds current = iter.next();
+boolean isRange = current instanceof Range;
+
+Token min = current.partitioner.getMinimumToken();
+while (iter.hasNext())
+{
+if (current.right.equals(min))
+{
+// If one of the bound is the full range, we return only that
+if (current.left.equals(min))
+return Collections.AbstractBoundssingletonList(current);
+
+output.add(current.createFrom(min));
+return output;
+}
+
+AbstractBounds next = iter.next();
+assert isRange ? next instanceof Range : next instanceof Bounds;
+
+// For Ranges, if next left is equal to current right, we do not
intersect per se, but replacing (A, B] and (B, C] by (A, C] is
+// legit, and since this actually avoid special casing and will
result in more optimal ranges, we do this transformation
+if (next.left.compareTo(current.right) = 0)
+{
+// We do overlap
+// (we've handler current.right.equals(min) already)
+Token newRight = next.right.equals(min) ||
current.right.compareTo(next.right) 0 ? next.right : current.right;
+current = current.createFrom(newRight);
+if (current == null)
+// current is the full ring, can only happen for Range
+return Collections.AbstractBoundssingletonList(new
Range(min, min));
+}
+else
+{
+output.add(current);
+current = next;
+}
+}
+output.add(current);
return output;
}
Added:
cassandra/trunk/test/unit/org/apache/cassandra/dht/AbstractBoundsTest.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/test/unit/org/apache/cassandra/dht/AbstractBoundsTest.java?rev=1136008view=auto
==
--- cassandra/trunk/test/unit/org/apache/cassandra/dht/AbstractBoundsTest.java
(added)
+++
[jira] [Resolved] (CASSANDRA-2641) AbstractBounds.normalize should deal with overlapping ranges
[ https://issues.apache.org/jira/browse/CASSANDRA-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne resolved CASSANDRA-2641. - Resolution: Fixed Fix Version/s: (was: 0.8.1) 1.0 Reviewer: stuhood (was: slebresne) Assignee: Sylvain Lebresne (was: Stu Hood) Committed to 1.0. Since I'm pretty sure we don't generate overlapping range so far, it's not worth taking the risk to put in 0.8. AbstractBounds.normalize should deal with overlapping ranges Key: CASSANDRA-2641 URL: https://issues.apache.org/jira/browse/CASSANDRA-2641 Project: Cassandra Issue Type: Test Components: Core Reporter: Stu Hood Assignee: Sylvain Lebresne Priority: Minor Fix For: 1.0 Attachments: 0001-Assert-non-overlapping-ranges-in-normalize.txt, 0001-Make-normalize-deoverlap-ranges.patch, 0002-Don-t-use-overlapping-ranges-in-tests.txt Apparently no consumers have encountered it in production, but AbstractBounds.normalize does not handle overlapping ranges. If given overlapping ranges, the output will be sorted but still overlapping, for which SSTableReader.getPositionsForRanges will choose ranges in an SSTable that may overlap. We should either add an assert in normalize(), or in getPositionsForRanges() to ensure that this never bites us in production. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0002-Register-in-gossip-to-handle-node-failures-v3.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0002-Register-in-gossip-to-handle-node-failures-v2.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re-v3.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re-v2.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0002-Register-in-gossip-to-handle-node-failures.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0004-Reports-validation-compaction-errors-back-to-repair-v2.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0004-Reports-validation-compaction-errors-back-to-repair.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0003-Report-streaming-errors-back-to-repair-v2.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: (was: 0003-Report-streaming-errors-back-to-repair-v3.patch) Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2433) Failed Streams Break Repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2433: Attachment: 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch 0003-Report-streaming-errors-back-to-repair-v4.patch 0002-Register-in-gossip-to-handle-node-failures-v4.patch 0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re-v4.patch Attaching v4 that is rebased and simply set the reties variable in StreamInSession volatile after all (I've removed old version because it was a mess). Failed Streams Break Repair --- Key: CASSANDRA-2433 URL: https://issues.apache.org/jira/browse/CASSANDRA-2433 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Labels: repair Fix For: 0.8.1 Attachments: 0001-Put-repair-session-on-a-Stage-and-add-a-method-to-re-v4.patch, 0002-Register-in-gossip-to-handle-node-failures-v4.patch, 0003-Report-streaming-errors-back-to-repair-v4.patch, 0004-Reports-validation-compaction-errors-back-to-repair-v4.patch Running repair in cases where a stream fails we are seeing multiple problems. 1. Although retry is initiated and completes, the old stream doesn't seem to clean itself up and repair hangs. 2. The temp files are left behind and multiple failures can end up filling up the data partition. These issues together are making repair very difficult for nearly everyone running repair on a non-trivial sized data set. This issue is also being worked on w.r.t CASSANDRA-2088, however that was moved to 0.8 for a few reasons. This ticket is to fix the immediate issues that we are seeing in 0.7. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2732) StringIndexOutOfBoundsException when specifying JDBC connection string without user and password
[
https://issues.apache.org/jira/browse/CASSANDRA-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049748#comment-13049748
]
Vivek Mishra commented on CASSANDRA-2732:
-
Does it mean specifiying something like cassandra.properties to hold these
parameters and loading them up with DriverManager. Will it make getting a jdbc
connection without even specifying a URI?
In this we can remove such checks for '@' or something like this.
As i believe username and password as part of connection parameters doesn't
make much sense with Cassandra.
StringIndexOutOfBoundsException when specifying JDBC connection string
without user and password
Key: CASSANDRA-2732
URL: https://issues.apache.org/jira/browse/CASSANDRA-2732
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8 beta 1
Reporter: Cathy Daw
Assignee: Rick Shaw
Priority: Trivial
Labels: cql
*PASS: specify connection string user and password*
_connection =
DriverManager.getConnection(jdbc:cassandra:root/root@localhost:9170/default)_
*FAIL: specify connection string without user and password*
_connection =
DriverManager.getConnection(jdbc:cassandra://localhost:9170/default)_
{code}
[junit] String index out of range: -1
[junit] java.lang.StringIndexOutOfBoundsException: String index out of range:
-1
[junit] at java.lang.String.substring(String.java:1937)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraConnection.init(CassandraConnection.java:74)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraConnection.init(CassandraConnection.java:74)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraDriver.connect(CassandraDriver.java:86)
[junit] at java.sql.DriverManager.getConnection(DriverManager.java:582)
[junit] at java.sql.DriverManager.getConnection(DriverManager.java:207)
[junit] at
com.datastax.cql.runJDBCSmokeTest.setUpBeforeClass(runJDBCSmokeTest.java:45)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2732) StringIndexOutOfBoundsException when specifying JDBC connection string without user and password
[
https://issues.apache.org/jira/browse/CASSANDRA-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049749#comment-13049749
]
Vivek Mishra commented on CASSANDRA-2732:
-
Specifying property set is supported by JDBC. But putting something like
cassandra.properties can be more useful as it will allow us to put more
configurable parameters may be needed in future, like batch flush, batch fetch
etc etc.
StringIndexOutOfBoundsException when specifying JDBC connection string
without user and password
Key: CASSANDRA-2732
URL: https://issues.apache.org/jira/browse/CASSANDRA-2732
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8 beta 1
Reporter: Cathy Daw
Assignee: Rick Shaw
Priority: Trivial
Labels: cql
*PASS: specify connection string user and password*
_connection =
DriverManager.getConnection(jdbc:cassandra:root/root@localhost:9170/default)_
*FAIL: specify connection string without user and password*
_connection =
DriverManager.getConnection(jdbc:cassandra://localhost:9170/default)_
{code}
[junit] String index out of range: -1
[junit] java.lang.StringIndexOutOfBoundsException: String index out of range:
-1
[junit] at java.lang.String.substring(String.java:1937)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraConnection.init(CassandraConnection.java:74)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraConnection.init(CassandraConnection.java:74)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraDriver.connect(CassandraDriver.java:86)
[junit] at java.sql.DriverManager.getConnection(DriverManager.java:582)
[junit] at java.sql.DriverManager.getConnection(DriverManager.java:207)
[junit] at
com.datastax.cql.runJDBCSmokeTest.setUpBeforeClass(runJDBCSmokeTest.java:45)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.
[
https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049724#comment-13049724
]
Mck SembWever edited comment on CASSANDRA-2388 at 6/15/11 12:46 PM:
bq. [snip] One possibility is to use the ip octets like the
RackInferringSnitch.
In our usecase we have three nodes defined via
PropertyFileSnitch:{noformat}152.90.241.22=DC1:RAC1 #node1
152.90.241.23=DC2:RAC1 #node2
152.90.241.24=DC1:RAC1 #node3{noformat}
When we make the connection through the other (node2) endpoint taking the
rack inferring approach 152.90. will say it's in DC2. (again) this is the
wrong DC and will return itself as a valid endpoint
Step (3) seems to me to be too specific to be included here.
If i go only with steps (1),(2),and (4) we get this code:{noformat}public
String[] sort_endpoints_by_proximity(String endpoint, String[] endpoints,
boolean restrictToSameDC)
throws TException, InvalidRequestException
{
try
{
ListString results = new ArrayListString();
InetAddress address = InetAddress.getByName(endpoint);
boolean endpointValid = null !=
Gossiper.instance.getEndpointStateForEndpoint(address);
String datacenter = DatabaseDescriptor
.getEndpointSnitch().getDatacenter(endpointValid ? address
: FBUtilities.getLocalAddress());
ListInetAddress addresses = new ArrayListInetAddress();
for(String ep : endpoints)
{
addresses.add(InetAddress.getByName(endpoint));
}
DatabaseDescriptor.getEndpointSnitch().sortByProximity(address,
addresses);
for(InetAddress ep : addresses)
{
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(ep);
if(FailureDetector.instance.isAlive(ep) (!restrictToSameDC
|| datacenter.equals(dc)))
{
results.add(ep.getHostName());
}
}
return results.toArray(new String[results.size()]);
}
catch (UnknownHostException e)
{
throw new InvalidRequestException(e.getMessage());
}
}{noformat}
I'm happy with this (except that
{{Gossiper.instance.getEndpointStateForEndpoint(address)}} is only my guess on
how to tell if an endpoint is valid as such).
was (Author: michaelsembwever):
bq. [snip] One possibility is to use the ip octets like the
RackInferringSnitch.
In our usecase we have three nodes defined via
PropertyFileSnitch:{noformat}152.90.241.22=DC1:RAC1 #node1
152.90.241.23=DC2:RAC1 #node2
152.90.241.24=DC1:RAC1 #node3{noformat}
When we make the connection through the other (node2) endpoint taking the
rack inferring approach 152.90. will say it's in DC2. (again) this is the
wrong DC and will return itself as a valid endpoint
Step (3) seems to me to be too specific to be included here.
If i take it out we get this code:{noformat}public String[]
sort_endpoints_by_proximity(String endpoint, String[] endpoints, boolean
restrictToSameDC)
throws TException, InvalidRequestException
{
try
{
ListString results = new ArrayListString();
InetAddress address = InetAddress.getByName(endpoint);
boolean endpointValid = null !=
Gossiper.instance.getEndpointStateForEndpoint(address);
String datacenter = DatabaseDescriptor
.getEndpointSnitch().getDatacenter(endpointValid ? address
: FBUtilities.getLocalAddress());
ListInetAddress addresses = new ArrayListInetAddress();
for(String ep : endpoints)
{
addresses.add(InetAddress.getByName(endpoint));
}
DatabaseDescriptor.getEndpointSnitch().sortByProximity(address,
addresses);
for(InetAddress ep : addresses)
{
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(ep);
if(FailureDetector.instance.isAlive(ep) (!restrictToSameDC
|| datacenter.equals(dc)))
{
results.add(ep.getHostName());
}
}
return results.toArray(new String[results.size()]);
}
catch (UnknownHostException e)
{
throw new InvalidRequestException(e.getMessage());
}
}{noformat}
I'm happy with this (except that
{{Gossiper.instance.getEndpointStateForEndpoint(address)}} is only my guess on
how to tell if an endpoint is valid as such).
ColumnFamilyRecordReader fails for a given split because a host is down, even
if records could reasonably be read from other replica.
-
[jira] [Commented] (CASSANDRA-2732) StringIndexOutOfBoundsException when specifying JDBC connection string without user and password
[
https://issues.apache.org/jira/browse/CASSANDRA-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049754#comment-13049754
]
Rick Shaw commented on CASSANDRA-2732:
--
With the new changes a special cassandra properties file will not be necessary.
You may now use the properties argument in the DriverManager (or the username
and password arguments) it is also fully supported in the factory methods. It
works with Datasources as well.
I agree that the URI is not the right place for user and password but it will
continue to be supported for convenience. The properties or the un/pw arguments
to the Drivermanager will alway take precedence.
Any additional helpful parameters to give hints to the driver are supposed to
be passed in this fashion. There just are not any that are supported yet. When
we support built in pooling (external pooling of connection works now) those
arguments you spoke of will be passed in the properties argument as well.
The goal is this should look and feel like any other JDBC driver for a DB with
the few limitations that are unique to C*.
StringIndexOutOfBoundsException when specifying JDBC connection string
without user and password
Key: CASSANDRA-2732
URL: https://issues.apache.org/jira/browse/CASSANDRA-2732
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8 beta 1
Reporter: Cathy Daw
Assignee: Rick Shaw
Priority: Trivial
Labels: cql
*PASS: specify connection string user and password*
_connection =
DriverManager.getConnection(jdbc:cassandra:root/root@localhost:9170/default)_
*FAIL: specify connection string without user and password*
_connection =
DriverManager.getConnection(jdbc:cassandra://localhost:9170/default)_
{code}
[junit] String index out of range: -1
[junit] java.lang.StringIndexOutOfBoundsException: String index out of range:
-1
[junit] at java.lang.String.substring(String.java:1937)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraConnection.init(CassandraConnection.java:74)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraConnection.init(CassandraConnection.java:74)
[junit] at
org.apache.cassandra.cql.jdbc.CassandraDriver.connect(CassandraDriver.java:86)
[junit] at java.sql.DriverManager.getConnection(DriverManager.java:582)
[junit] at java.sql.DriverManager.getConnection(DriverManager.java:207)
[junit] at
com.datastax.cql.runJDBCSmokeTest.setUpBeforeClass(runJDBCSmokeTest.java:45)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
[ https://issues.apache.org/jira/browse/CASSANDRA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049770#comment-13049770 ] Sylvain Lebresne commented on CASSANDRA-2405: - The problem with using the completion time as the (Super)Column name is that you have to wait the end of the repair to store anything. First, this will not capture started but failed session (which while not mandatory could be nice, especially as soon as we will start keeping a bit more info this could help troubleshooting). And Second, it will be a pain to have to keep some of the information until the end (the processingStartedAt is a first sign of this). And third, we may want to keep some info on say merkle tree creation on all replica participating in the repair, even though we only store the completed time on the node initiating the repair. So I would propose to something like: row key: KS/CF super column name: repair session name (a TimeUUID) columns: the infos on the session (range, start and end time, number of range repaired, bytes transferred, ...) That is roughly the same thing as you propose but with super column name being the repair session name. Now, because the repair session names are TimeUUID (well, right now it is a sting including a UUID, we can change it to a simple TimeUUID easily), the session will be ordered by creation time. So getting the last successful repair is probably not too hard: just grab the last 1000 created sessions and find the last successful one. And if we want, we can even use another specific index row that associate 'completion time' - 'session UUID' (and thanks to the new DynamicCompositeType we can have some rows ordered by TimeUUIDType and some other ordered by LongType without the need of multiple system table). should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Peter Schuller Assignee: Pavel Yaskevich Priority: Minor Fix For: 0.8.1 Attachments: CASSANDRA-2405-v2.patch, CASSANDRA-2405-v3.patch, CASSANDRA-2405.patch The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049771#comment-13049771 ] Yang Yang commented on CASSANDRA-2774: -- --- the only valid answer not really In fact this is the same reasoning that I explained in your Net partition case on gmail.g list You expect that it is the only valid case* * Because* ** of assumed timing order from client respective but we know that timing Is not trustworthy and this order on client is Not respected because client does not participate In the logical clock protocol of epoch generation Btw we'd better force ROW for delete so we achieve Faster agreement and lose fewer adds On old epoch. I think with quorum delete you will guarantee timing to be consistent eoyh client And then achieve client expected result I. Your Case, id like to hear your counter example Pardon the typing, on phone, will be On computer soon one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
[ https://issues.apache.org/jira/browse/CASSANDRA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049775#comment-13049775 ] Pavel Yaskevich commented on CASSANDRA-2405: We need to decide - are we going with description of this task or trying to build logging for repair? should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Peter Schuller Assignee: Pavel Yaskevich Priority: Minor Fix For: 0.8.1 Attachments: CASSANDRA-2405-v2.patch, CASSANDRA-2405-v3.patch, CASSANDRA-2405.patch The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
[ https://issues.apache.org/jira/browse/CASSANDRA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049784#comment-13049784 ] Sylvain Lebresne commented on CASSANDRA-2405: - I'm keen on adding persisted stats for repair for CASSANDRA-2698. Recording the start and end time of repair also amounts to persisting stats on repair. Given that, I don't care too much about what the description of this says, but I'm pretty much opposed to doing anything here that would make CASSANDRA-2698 much harder that it needs unless there is a good reason, and I don't see one. I'm happy with making this a duplicate or dependency of CASSANDRA-2698 though. should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Peter Schuller Assignee: Pavel Yaskevich Priority: Minor Fix For: 0.8.1 Attachments: CASSANDRA-2405-v2.patch, CASSANDRA-2405-v3.patch, CASSANDRA-2405.patch The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049805#comment-13049805 ] Yang Yang commented on CASSANDRA-2774: -- It's good to step back and ask what is the contract ? Now So far cassandra let time be dictated arbitrarily by client ( Client assigns timestamp) so it can afford to expect* the* *** Ordering and *the** result In counters client-generated ts can't be used so we can't guarantee* the* expected Result. But we will guaranttee that all nodes on the system eventually agree on *some* common result, this is actually consistent with the definition Of eventual consistency brought forth by Dynamo ( if u look at client-generated ts. It could indeed produce some Unintuitive results: if a client sets ts to be very late into The future, latter writes could be wiped out. For regular columns This is in the same way against common client expectations about ordering as in the counter case) In summary the current implementation cannot guarattee any common agreement on counter value but the new approach guarantees thecommon value Achievement of some one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2761) JDBC driver does not build
[ https://issues.apache.org/jira/browse/CASSANDRA-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rick Shaw updated CASSANDRA-2761: - Attachment: jdbc-driver-build-v2.txt The v2 patch adds inclusion of classes that are required on the client side from the main C* build. JDBC driver does not build -- Key: CASSANDRA-2761 URL: https://issues.apache.org/jira/browse/CASSANDRA-2761 Project: Cassandra Issue Type: Bug Components: API Affects Versions: 1.0 Reporter: Jonathan Ellis Assignee: Rick Shaw Fix For: 1.0 Attachments: jdbc-driver-build-v1.txt, jdbc-driver-build-v2.txt Need a way to build (and run tests for) the Java driver. Also: still some vestigal references to drivers/ in trunk build.xml. Should we remove drivers/ from the 0.8 branch as well? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049805#comment-13049805 ] Yang Yang edited comment on CASSANDRA-2774 at 6/15/11 3:48 PM: --- It's good to step back and ask what is the contract ? Now So far cassandra let time be dictated arbitrarily by client ( Client assigns timestamp) so it can afford to expect *the* Ordering and *the* result In counters client-generated ts can't be used so we can't guarantee *the* expected Result. But we will guaranttee that all nodes on the system eventually agree on *some* common result, this is actually consistent with the definition Of eventual consistency brought forth by Dynamo ( if u look at client-generated ts. It could indeed produce some Unintuitive results: if a client sets ts to be very late into The future, latter writes could be wiped out. For regular columns This is in the same way against common client expectations about ordering as in the counter case) In summary the current implementation cannot guarattee any common agreement on counter value but the new approach guarantees the achievement of *some* common value was (Author: yangyangyyy): It's good to step back and ask what is the contract ? Now So far cassandra let time be dictated arbitrarily by client ( Client assigns timestamp) so it can afford to expect* the* *** Ordering and *the** result In counters client-generated ts can't be used so we can't guarantee* the* expected Result. But we will guaranttee that all nodes on the system eventually agree on *some* common result, this is actually consistent with the definition Of eventual consistency brought forth by Dynamo ( if u look at client-generated ts. It could indeed produce some Unintuitive results: if a client sets ts to be very late into The future, latter writes could be wiped out. For regular columns This is in the same way against common client expectations about ordering as in the counter case) In summary the current implementation cannot guarattee any common agreement on counter value but the new approach guarantees thecommon value Achievement of some one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049771#comment-13049771 ] Yang Yang edited comment on CASSANDRA-2774 at 6/15/11 3:51 PM: --- --- the only valid answer not really In fact this is the same reasoning that I explained in your Net partition case on gmail.g list You expect that it is the only valid case *because* of assumed timing order from client respective but we know that timing Is not trustworthy and this order on client is Not respected because client does not participate In the logical clock protocol of epoch generation Btw we'd better force ROW for delete so we achieve Faster agreement and lose fewer adds On old epoch. I think with quorum delete you will guarantee timing to be consistent with client And then achieve client expected result I. Your Case, id like to hear your counter example Pardon the typing, on phone, will be On computer soon was (Author: yangyangyyy): --- the only valid answer not really In fact this is the same reasoning that I explained in your Net partition case on gmail.g list You expect that it is the only valid case* * Because* ** of assumed timing order from client respective but we know that timing Is not trustworthy and this order on client is Not respected because client does not participate In the logical clock protocol of epoch generation Btw we'd better force ROW for delete so we achieve Faster agreement and lose fewer adds On old epoch. I think with quorum delete you will guarantee timing to be consistent eoyh client And then achieve client expected result I. Your Case, id like to hear your counter example Pardon the typing, on phone, will be On computer soon one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049805#comment-13049805 ] Yang Yang edited comment on CASSANDRA-2774 at 6/15/11 3:49 PM: --- It's good to step back and ask what is the contract ? now. So far cassandra lets time be dictated arbitrarily by client ( Client assigns timestamp) so it can afford to expect *the* Ordering and *the* result In counters client-generated ts can't be used so we can't guarantee *the* expected Result. But we will guaranttee that all nodes on the system eventually agree on *some* common result, this is actually consistent with the definition Of eventual consistency brought forth by Dynamo ( if u look at client-generated ts. It could indeed produce some Unintuitive results: if a client sets ts to be very late into The future, latter writes could be wiped out. For regular columns This is in the same way against common client expectations about ordering as in the counter case) In summary the current implementation cannot guarattee any common agreement on counter value but the new approach guarantees the achievement of *some* common value was (Author: yangyangyyy): It's good to step back and ask what is the contract ? Now So far cassandra let time be dictated arbitrarily by client ( Client assigns timestamp) so it can afford to expect *the* Ordering and *the* result In counters client-generated ts can't be used so we can't guarantee *the* expected Result. But we will guaranttee that all nodes on the system eventually agree on *some* common result, this is actually consistent with the definition Of eventual consistency brought forth by Dynamo ( if u look at client-generated ts. It could indeed produce some Unintuitive results: if a client sets ts to be very late into The future, latter writes could be wiped out. For regular columns This is in the same way against common client expectations about ordering as in the counter case) In summary the current implementation cannot guarattee any common agreement on counter value but the new approach guarantees the achievement of *some* common value one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049834#comment-13049834 ] Sylvain Lebresne commented on CASSANDRA-2774: - bq. I think with quorum delete you will guarantee timing to be consistent eoyh client And then achieve client expected result I. Your Case, id like to hear your counter example Consider a cluster with RF=3 and counter c replicated on node A, B and C. Consider that all operation are done by the same client connected to some other node (doesn't have to be the same each time but can be). All operations are performed at QUORUM consistency level. The client does the following operations: # increment c by 1 # delete c # increment c by 1 # reads c Because QUORUM is 2, depending on internal timings (latency on the wire and such), either only 2 or the 3 nodes will have seen each write once it is acked to the client. Again, for the same inputs and depending on timing, the client could get on the read a variety of results: * 1 if each node have received each operation in the order issued. * 0 or 2, if for instance, by the time the read is issued: ** the first increment only reached B and C ** the deletion only reached A and C ** the second increment only reached A and B and it happens that the two first node answering the read are B and C. The exact value depends on the exact rules for dealing with the epoch number, but in any case, B would only have the two increments and C would have the first increment and deletion (issued after the increment, so the deletion wins). So B will answer 2 and C will answer a tombstone. Whatever resolution the coordinator does, it just cannot return 1 that time. one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2369) support replication decisions per-key
[
https://issues.apache.org/jira/browse/CASSANDRA-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049855#comment-13049855
]
Jonathan Ellis commented on CASSANDRA-2369:
---
No, it's worse than that.
Let me give an example of a simple multi-node, multi-DC cluster: nodes A and M
in DC1, nodes B and N in DC2. So node A, M, B, and N have keys in ranges
(M-A], (A, M], (N, B], (B, N], respectively.
If I write a row K with NTS {DC1: 1, DC2:2}, then I know it will be on nodes M
and N. So far so good.
What if I now repair node M? It knows it has to compare its data for range (A,
B] with that data on node B, and range (B, M] with that data on node N. So it
builds a merkle tree for each range, and requests that B and N do so as well,
then they exchange trees to see if things are in sync.
How does this change if we introduce this partitioner? M can no longer assume
that keys it has for range (A, B] should also be replicated to node M, and vice
versa. You would have to build a separate tree for each replica, i.e. instead
of just a tree for (A, B], each replica would need to build a tree for (A,
B]-that-belongs-on-M, and another tree for (A, B)-that-belongs-on-B, and so
forth for as many possible replicas as exist.
There is a similar problem on bootstrap and node movement. Instead of asking a
single replica to stream data from the range a new node is assuming, it will
have to ask _all_ replicas that may have rows for that range to make sure it
doesn't miss any.
support replication decisions per-key
-
Key: CASSANDRA-2369
URL: https://issues.apache.org/jira/browse/CASSANDRA-2369
Project: Cassandra
Issue Type: New Feature
Reporter: Jonathan Ellis
Assignee: Vijay
Priority: Minor
Fix For: 1.0
Currently the replicationstrategy gets a token and a keyspace with which to
decide how to place replicas. for per-row replication this is insufficient
because tokenization is lossy (CASSANDRA-1034).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
[ https://issues.apache.org/jira/browse/CASSANDRA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049867#comment-13049867 ] Pavel Yaskevich commented on CASSANDRA-2405: I'm thinking of using range as supercolumn and storing session_id, start/completion times as subcolumns, that will give us possibility to log individually each of the repairs made in the session. should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Peter Schuller Assignee: Pavel Yaskevich Priority: Minor Fix For: 0.8.1 Attachments: CASSANDRA-2405-v2.patch, CASSANDRA-2405-v3.patch, CASSANDRA-2405.patch The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049874#comment-13049874 ] Yang Yang commented on CASSANDRA-2774: -- you are right it just cannot return 1 (at) *that time* , 0 or 2 is the value not stable that the system had from some past snapshot in time. but it will eventually come to answer 1: since our edge case above assumes that B has not got the deletion yet, the leader in the second increment can not be A, cuz otherwise B must have got the deletion from A, since on A the increment comes later. so B was the leader in the second increment. for C, it now has new epoch, let's say A's second increment reaches C (through repair, since A is not the leader in second increment), this increment has new epoch, so it will be accepted by C; if B's second increment reaches C, it belongs to the old epoch, it will be rejected. for B, it is still on the old epoch, after the second increment, B's count is 2 of the old epoch. but when A's increment goes to B through repair, or is reconciled in read with B, the result is going to be 1. if C's deletion goes to B, B is going to be brought more up to date to a value of 0 of new epoch. the above analysis does not go through all possible scenarios, but to give a definitive proof of the conjecture that all nodes return *the* ordering given by client , in case of quorum read/write, I need to think more. but as I stated in my last comment, at least we can be sure that the new approach guarantees *some* common agreement eventually. it would be nice if we achieve *the* agreement in case of quorum, but that's not my main argument one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2369) support replication decisions per-key
[ https://issues.apache.org/jira/browse/CASSANDRA-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049871#comment-13049871 ] Vijay commented on CASSANDRA-2369: -- Hi Jonathan, Yeah, i started coding the repair case (1 to 1 repair, something similar to what you mentioned above) quickly realized the complexity it buys... than any good... Max replication can help with the boot strap and repair (but wont be efficient at per sure)... support replication decisions per-key - Key: CASSANDRA-2369 URL: https://issues.apache.org/jira/browse/CASSANDRA-2369 Project: Cassandra Issue Type: New Feature Reporter: Jonathan Ellis Assignee: Vijay Priority: Minor Fix For: 1.0 Currently the replicationstrategy gets a token and a keyspace with which to decide how to place replicas. for per-row replication this is insufficient because tokenization is lossy (CASSANDRA-1034). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2614) create Column and CounterColumn in the same column family
[ https://issues.apache.org/jira/browse/CASSANDRA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049877#comment-13049877 ] Jonathan Ellis commented on CASSANDRA-2614: --- Not worth breaking Thrift compatibility means not in 1.0 either. create Column and CounterColumn in the same column family - Key: CASSANDRA-2614 URL: https://issues.apache.org/jira/browse/CASSANDRA-2614 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Dave Rav Assignee: Sylvain Lebresne Priority: Minor Fix For: 0.8.1 create Column and CounterColumn in the same column family -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
[ https://issues.apache.org/jira/browse/CASSANDRA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049882#comment-13049882 ] Sylvain Lebresne commented on CASSANDRA-2405: - Hum, the thing is that there will be many repair sessions for a given set of KS/CF and range. So you need one of the key (either row key or supercolumn name) to be the session_id (or anything that is unique to a session). If you use a row for each KS/CF pair and one super column for each session, you will have one super column for each repair made in a session (or kind of, you will indeed have multiple merkle tree for instance, one for each replica, but we can easily prefix the column with the replica name if need be). should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Peter Schuller Assignee: Pavel Yaskevich Priority: Minor Fix For: 0.8.1 Attachments: CASSANDRA-2405-v2.patch, CASSANDRA-2405-v3.patch, CASSANDRA-2405.patch The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2369) support replication decisions per-key
[ https://issues.apache.org/jira/browse/CASSANDRA-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049899#comment-13049899 ] Sylvain Lebresne commented on CASSANDRA-2369: - Let me also add that if you allow that, load balancing will be a bitch. One may argue it should be the problem of whomever wants to use this, but I'm not sure that providing tools that make foot-shooting too easy is such a good idea. support replication decisions per-key - Key: CASSANDRA-2369 URL: https://issues.apache.org/jira/browse/CASSANDRA-2369 Project: Cassandra Issue Type: New Feature Reporter: Jonathan Ellis Assignee: Vijay Priority: Minor Fix For: 1.0 Currently the replicationstrategy gets a token and a keyspace with which to decide how to place replicas. for per-row replication this is insufficient because tokenization is lossy (CASSANDRA-1034). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049912#comment-13049912 ] Sylvain Lebresne commented on CASSANDRA-2774: - bq. but as I stated in my last comment, at least we can be sure that the new approach guarantees some common agreement eventually. It is already the case with the current implementation. Once compaction has compacted the deletes, all node will reach common agreement. bq. it would be nice if we achieve the agreement in case of quorum, but that's not my main argument My main argument is that this patch slightly change the behavior here and there but I don't think it adds any tangible new guarantee that people can work with. On the other side, it adds a fairly heavy performance hit by adding a read before write on every replica (and though you won't necessary do a read for every write, you will do that read more often than not as soon as the set of counters you're incrementing is not small enough). one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1136133 - /cassandra/branches/cassandra-0.8/test/system/test_cql.py
Author: jbellis Date: Wed Jun 15 17:45:00 2011 New Revision: 1136133 URL: http://svn.apache.org/viewvc?rev=1136133view=rev Log: r/m unused import Modified: cassandra/branches/cassandra-0.8/test/system/test_cql.py Modified: cassandra/branches/cassandra-0.8/test/system/test_cql.py URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/test/system/test_cql.py?rev=1136133r1=1136132r2=1136133view=diff == --- cassandra/branches/cassandra-0.8/test/system/test_cql.py (original) +++ cassandra/branches/cassandra-0.8/test/system/test_cql.py Wed Jun 15 17:45:00 2011 @@ -24,7 +24,6 @@ import sys, uuid, time sys.path.append(join(abspath(dirname(__file__)), '../../drivers/py')) import cql -from cql.connection import Connection from __init__ import ThriftTester from __init__ import thrift_client # TODO: temporary
[Cassandra Wiki] Trivial Update of ThirdPartySupport by JoseAvila
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Cassandra Wiki for
change notification.
The ThirdPartySupport page has been changed by JoseAvila:
http://wiki.apache.org/cassandra/ThirdPartySupport?action=diffrev1=8rev2=9
Comment:
Updating company description
Companies providing support for Apache Cassandra are not endorsed by the
Apache Software Foundation, although some of these companies employ
[[Committers]] to the Apache project.
{{http://www.datastax.com/sites/all/themes/datastax20110201/logo.png}}
[[http://datastax.com|Datastax]] provides professional Cassandra support and
services.
+
+ {{http://www.onzra.com/images/Small-Logo.gif}} [[http://www.ONZRA.com|ONZRA]]
has been around for over 10 years and specializes on enterprise grade
architecture, development and security consulting services utilizing many large
scale database technologies such as Cassandra, Oracle, AlegroGraph, and much
more.
{{http://acunu.com/images/logo.png}} [[http://www.acunu.com|Acunu]] provides
software products for Cassandra applications, as well as Cassandra support and
professional services.
@@ -10, +12 @@
MongoDB, and Other databases like Riak, Redis, Membase, Tokyocabinet, etc
[[http://bigdata.impetus.com/# | More info about BigData @Impetus]]
- {{http://www.onzra.com/images/Small-Logo.gif}} [[http://www.ONZRA.com|ONZRA]]
provides enterprise grade Cassandra architecture and development services.
[Cassandra Wiki] Trivial Update of ThirdPartySupport by JoseAvila
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Cassandra Wiki for
change notification.
The ThirdPartySupport page has been changed by JoseAvila:
http://wiki.apache.org/cassandra/ThirdPartySupport?action=diffrev1=9rev2=10
{{http://www.datastax.com/sites/all/themes/datastax20110201/logo.png}}
[[http://datastax.com|Datastax]] provides professional Cassandra support and
services.
- {{http://www.onzra.com/images/Small-Logo.gif}} [[http://www.ONZRA.com|ONZRA]]
has been around for over 10 years and specializes on enterprise grade
architecture, development and security consulting services utilizing many large
scale database technologies such as Cassandra, Oracle, AlegroGraph, and much
more.
+ {{http://www.onzra.com/images/Small-Logo.gif}} [[http://www.ONZRA.com|ONZRA]]
has been around for over 10 years and specializes on enterprise grade
architecture, development and security consulting services utilizing many large
scale database technologies such as Cassandra, Oracle, Alegro Graph, and much
more.
{{http://acunu.com/images/logo.png}} [[http://www.acunu.com|Acunu]] provides
software products for Cassandra applications, as well as Cassandra support and
professional services.
[jira] [Commented] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049919#comment-13049919 ] Jonathan Ellis commented on CASSANDRA-2769: --- the 0.8 patch looks good. (i did notice that some of the other post-markCompactiong code checks for null or empty, others just check for null -- one of these is probably wrong.) Cannot Create Duplicate Compaction Marker - Key: CASSANDRA-2769 URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.0 Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Fix For: 0.8.1, 1.0 Attachments: 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch, 0001-Do-compact-only-smallerSSTables.patch, 0002-Only-compact-what-has-been-succesfully-marked-as-com.patch Concurrent compaction can trigger the following exception when two threads compact the same sstable. DataTracker attempts to prevent this but apparently not successfully. java.io.IOError: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1136135 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/db/compaction/CompactionManager.java
Author: slebresne
Date: Wed Jun 15 17:55:31 2011
New Revision: 1136135
URL: http://svn.apache.org/viewvc?rev=1136135view=rev
Log:
Avoids unmarking compacting sstable prematurely during cleanup
patch by slebresne; reviewed by jbellis for CASSANDRA-2769
Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1136135r1=1136134r2=1136135view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Wed Jun 15 17:55:31 2011
@@ -57,6 +57,7 @@
(CASSANDRA-2767)
* use threadsafe collections for StreamInSession (CASSANDRA-2766)
* avoid infinite loop when creating merkle tree (CASSANDRA-2758)
+ * avoids unmarking compacting sstable prematurely in cleanup (CASSANDRA-2769)
0.8.0-final
Modified:
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java?rev=1136135r1=1136134r2=1136135view=diff
==
---
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
(original)
+++
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
Wed Jun 15 17:55:31 2011
@@ -845,56 +845,50 @@ public class CompactionManager implement
logger.debug(Expected bloom filter size : +
expectedBloomFilterSize);
SSTableWriter writer = null;
+
+logger.info(Cleaning up + sstable);
+// Calculate the expected compacted filesize
+long expectedRangeFileSize =
cfs.getExpectedCompactedFileSize(Arrays.asList(sstable)) / 2;
+String compactionFileLocation =
table.getDataFileLocation(expectedRangeFileSize);
+if (compactionFileLocation == null)
+throw new IOException(disk full);
+
+SSTableScanner scanner =
sstable.getDirectScanner(CompactionIterator.FILE_BUFFER_SIZE);
+SortedSetByteBuffer indexedColumns = cfs.getIndexedColumns();
+CleanupInfo ci = new CleanupInfo(sstable, scanner);
+executor.beginCompaction(ci);
try
{
-logger.info(Cleaning up + sstable);
-// Calculate the expected compacted filesize
-long expectedRangeFileSize =
cfs.getExpectedCompactedFileSize(Arrays.asList(sstable)) / 2;
-String compactionFileLocation =
table.getDataFileLocation(expectedRangeFileSize);
-if (compactionFileLocation == null)
-throw new IOException(disk full);
-
-SSTableScanner scanner =
sstable.getDirectScanner(CompactionIterator.FILE_BUFFER_SIZE);
-SortedSetByteBuffer indexedColumns = cfs.getIndexedColumns();
-CleanupInfo ci = new CleanupInfo(sstable, scanner);
-executor.beginCompaction(ci);
-try
+while (scanner.hasNext())
{
-while (scanner.hasNext())
+SSTableIdentityIterator row = (SSTableIdentityIterator)
scanner.next();
+if (Range.isTokenInRanges(row.getKey().token, ranges))
{
-SSTableIdentityIterator row =
(SSTableIdentityIterator) scanner.next();
-if (Range.isTokenInRanges(row.getKey().token, ranges))
-{
-writer = maybeCreateWriter(cfs,
compactionFileLocation, expectedBloomFilterSize, writer,
Collections.singletonList(sstable));
-writer.append(controller.getCompactedRow(row));
-totalkeysWritten++;
-}
-else
+writer = maybeCreateWriter(cfs,
compactionFileLocation, expectedBloomFilterSize, writer,
Collections.singletonList(sstable));
+writer.append(controller.getCompactedRow(row));
+totalkeysWritten++;
+}
+else
+{
+cfs.invalidateCachedRow(row.getKey());
+if (!indexedColumns.isEmpty() || isCommutative)
{
-cfs.invalidateCachedRow(row.getKey());
-if (!indexedColumns.isEmpty() || isCommutative)
+while (row.hasNext())
{
-while
[jira] [Commented] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049923#comment-13049923 ] Sylvain Lebresne commented on CASSANDRA-2769: - Alright, I've committed the 0.8 patch. I'll have a look at the checks. Cannot Create Duplicate Compaction Marker - Key: CASSANDRA-2769 URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.0 Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Fix For: 0.8.1, 1.0 Attachments: 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch, 0001-Do-compact-only-smallerSSTables.patch, 0002-Only-compact-what-has-been-succesfully-marked-as-com.patch Concurrent compaction can trigger the following exception when two threads compact the same sstable. DataTracker attempts to prevent this but apparently not successfully. java.io.IOError: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2775) ant build-test is failing with 0.8.0
[ https://issues.apache.org/jira/browse/CASSANDRA-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049934#comment-13049934 ] Vivek Mishra commented on CASSANDRA-2775: - Indirectly yes. But changes which i am discussing are just to make build-test working with current cassandra release(without releasing driver seperatly). Same set of change is required for generate-eclipse-files ant build-test is failing with 0.8.0 Key: CASSANDRA-2775 URL: https://issues.apache.org/jira/browse/CASSANDRA-2775 Project: Cassandra Issue Type: Bug Components: Tests Affects Versions: 0.8.0 Reporter: Vivek Mishra Assignee: Vivek Mishra Priority: Minor Attachments: build.xml Steps to reproduce: 1) ant clean build 2) ant build-test It is failing for : build-test: [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build.xml:975: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set [javac] Compiling 124 source files to C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\build\test\classes [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] CassandraPreparedStatement stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:357: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] CassandraPreparedStatement stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac]^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:365: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] C:\vivek\apache-cassandra-0.8.0-src\apache-cassandra-0.8.0-src\drivers\java\test\org\apache\cassandra\cql\jdbc\PreparedStatementTest.java:377: cannot find symbol [javac] symbol : class CassandraPreparedStatement [javac] location: class org.apache.cassandra.cql.jdbc.PreparedStatementTest [javac] stmt = (CassandraPreparedStatement)con.prepareStatement(q); [javac] ^ [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 4 errors -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2477) CQL support for describing keyspaces / column familes
[ https://issues.apache.org/jira/browse/CASSANDRA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049945#comment-13049945 ] Vivek Mishra commented on CASSANDRA-2477: - Absolutly.. make more sense. CQL support for describing keyspaces / column familes - Key: CASSANDRA-2477 URL: https://issues.apache.org/jira/browse/CASSANDRA-2477 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Eric Evans Labels: cql Fix For: 0.8.1 Attachments: 2477-virtual-cfs-false-start.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2629) Move key reads into SSTableIterators
[ https://issues.apache.org/jira/browse/CASSANDRA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stu Hood updated CASSANDRA-2629: Attachment: (was: 0001-CASSANDRA-2629-Move-key-and-row-size-reading-into-the-.txt) Move key reads into SSTableIterators Key: CASSANDRA-2629 URL: https://issues.apache.org/jira/browse/CASSANDRA-2629 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Stu Hood Fix For: 1.0 All SSTableIterators have a constructor that assumes the key and length has already been parsed. Moving this logic inside the iterator will improve symmetry and allow the file format to change without iterator consumers knowing it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2629) Move key reads into SSTableIterators
[ https://issues.apache.org/jira/browse/CASSANDRA-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stu Hood updated CASSANDRA-2629: Attachment: 0002-CASSANDRA-2629-Remove-the-retry-with-key-from-index-st.txt 0001-CASSANDRA-2629-Move-key-and-row-size-reading-into-the-.txt bq. move long rowSizeFromIndex = nextRowPositionFromIndex - currentRowPositionFromIndex; into the IF statement where it is needed Done. bq. in your log warnings, specifying the actual sstable will help with debugging Scrub starts with a log entry to indicate which file is being scrubbed... this should be sufficient unless someone is running without INFO. bq. remove this.key = key; for both constructors Good catch... that line was a noop in the second constructor: we don't know the key until init() is called, so we can't make it final. bq. *init() method should be more descriptive Was supposed to say Reads the key and row length from the head of the row.: fixed. bq. remove @param key comment from IFilter.java and SSTableSliceIterator.java Fixed. bq. safer to updateCache(iter) AFTER appending to writer I think the whole thing is racey, because the entire rebuilding process can fail, which might mean that the cache was updated when it shouldn't have been. I didn't make the close() change: we talked about close a little bit offline, I think we reached the consensus that having two methods that you must always call won't be an improvement over close. I also didn't add back the scrub retry: I agree that we can do better there, but I think it will be easier to improve once we have the checksumming from 674: at the moment the reasons for doing certain things with the API would not be clear at all. Thanks for the review! Move key reads into SSTableIterators Key: CASSANDRA-2629 URL: https://issues.apache.org/jira/browse/CASSANDRA-2629 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Stu Hood Fix For: 1.0 Attachments: 0001-CASSANDRA-2629-Move-key-and-row-size-reading-into-the-.txt, 0002-CASSANDRA-2629-Remove-the-retry-with-key-from-index-st.txt All SSTableIterators have a constructor that assumes the key and length has already been parsed. Moving this logic inside the iterator will improve symmetry and allow the file format to change without iterator consumers knowing it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
[ https://issues.apache.org/jira/browse/CASSANDRA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049958#comment-13049958 ] Pavel Yaskevich commented on CASSANDRA-2405: I suggest to create 2 CFs: one as we have right now so users can easily trigger time since last successful repair and CF to hold detailed information about each repair session with the following structure: sessionId (key), range (super column), subcolumns: started_at (long), completed_at (long), failed (0 or 1) should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Peter Schuller Assignee: Pavel Yaskevich Priority: Minor Fix For: 0.8.1 Attachments: CASSANDRA-2405-v2.patch, CASSANDRA-2405-v3.patch, CASSANDRA-2405.patch The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2477) CQL support for describing keyspaces / column familes
[ https://issues.apache.org/jira/browse/CASSANDRA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049961#comment-13049961 ] Vivek Mishra commented on CASSANDRA-2477: - From KsMetadata, something which i can see to get CFMetadata, strategy_class and strategy options... Count on number of ColumnFamilies, fetch specific CfMetadata(via cfName) Similarly from CFMetadata, we could fetch CfName, Default validator, compare type, subcolumn comparator, Keyspace name and many more.. Is this what we are planning for Describe Ks or Describe cf? CQL support for describing keyspaces / column familes - Key: CASSANDRA-2477 URL: https://issues.apache.org/jira/browse/CASSANDRA-2477 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Eric Evans Labels: cql Fix For: 0.8.1 Attachments: 2477-virtual-cfs-false-start.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2405) should expose 'time since last successful repair' for easier aes monitoring
[ https://issues.apache.org/jira/browse/CASSANDRA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049962#comment-13049962 ] Sylvain Lebresne commented on CASSANDRA-2405: - Nothing against that, though if we're going to have only a handful of rows in each it could be more efficient/cleaner to use the DynamicCompositeType instead of the creating two different CFs. Though if you absolutely prefer 2 CFs I won't fight against it. should expose 'time since last successful repair' for easier aes monitoring --- Key: CASSANDRA-2405 URL: https://issues.apache.org/jira/browse/CASSANDRA-2405 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Peter Schuller Assignee: Pavel Yaskevich Priority: Minor Fix For: 0.8.1 Attachments: CASSANDRA-2405-v2.patch, CASSANDRA-2405-v3.patch, CASSANDRA-2405.patch The practical implementation issues of actually ensuring repair runs is somewhat of an undocumented/untreated issue. One hopefully low hanging fruit would be to at least expose the time since last successful repair for a particular column family, to make it easier to write a correct script to monitor for lack of repair in a non-buggy fashion. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2761) JDBC driver does not build
[ https://issues.apache.org/jira/browse/CASSANDRA-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049983#comment-13049983 ] Rick Shaw commented on CASSANDRA-2761: -- I put together a patch that covered the obvious but I had strange problems in test. So I dug deeper and found a ton (over 50 I think) additional transitive dependencies that I had missed... (silly me) I can try an get them all in but I think we should seriously rethink that strategy for now. With this many dependancies they should probably be put in their own jar(s). It would really make the driver jar have to keep up with detailed dependency changes in the server code base. Miss one and it's messy and the errors are non-obvious. I was a big whiner about not carrying around the whole server just for client access, but until it is re-factored, I can now appreciate the horror that will commence if we piece-meal drag over classes from the server into the JAR. So fo now I suggest we keep the jar the way it is with just the o.a.c.cql.jdbc.* classes. JDBC driver does not build -- Key: CASSANDRA-2761 URL: https://issues.apache.org/jira/browse/CASSANDRA-2761 Project: Cassandra Issue Type: Bug Components: API Affects Versions: 1.0 Reporter: Jonathan Ellis Assignee: Rick Shaw Fix For: 1.0 Attachments: jdbc-driver-build-v1.txt Need a way to build (and run tests for) the Java driver. Also: still some vestigal references to drivers/ in trunk build.xml. Should we remove drivers/ from the 0.8 branch as well? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2681) Upgrade ConcurrentLinkedHashMap (v1.2)
[ https://issues.apache.org/jira/browse/CASSANDRA-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Burroughs reopened CASSANDRA-2681: Need to update in build.xml for the generated pom as well. Upgrade ConcurrentLinkedHashMap (v1.2) -- Key: CASSANDRA-2681 URL: https://issues.apache.org/jira/browse/CASSANDRA-2681 Project: Cassandra Issue Type: Task Components: Core Reporter: Ben Manes Assignee: Jonathan Ellis Priority: Minor Fix For: 1.0 This release has numerous performance improvements. See the change log for details. It also includes a few useful features that may be of interest, - Snapshot iteration in order of hotness (CASSANDRA-1966) - Optionally defer LRU maintenance penalty to a background executor (instead of amortized on caller threads) (Note that this setting is not advised if write storms are not rate limited, since it defers eviction until the executor runs.) http://code.google.com/p/concurrentlinkedhashmap/ http://code.google.com/p/concurrentlinkedhashmap/wiki/ExampleUsage Verified compatibility with NonBlockingHashMap. Cassandra may want to consider adding the java_util_concurrent_chm.jar to the bootclasspath to swap all CHM usages with NBHM (CLHM is a decorator on top of CHM). http://high-scale-lib.cvs.sourceforge.net/viewvc/high-scale-lib/high-scale-lib/README -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2477) CQL support for describing keyspaces / column familes
[ https://issues.apache.org/jira/browse/CASSANDRA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1304#comment-1304 ] Rick Shaw commented on CASSANDRA-2477: -- How about a custom CF for each (KS/CF); that fully defines the schema entries for each piece of information to be returned about each KS and each CF. Then DESCRIBE returns a CQLResult containing normal row content. The syntax of the DESCRIBE drives a flat set of rows returned to the caller. The CQL QueryProcessor returns the contents of the CQLRow items for information it already has access to in the internal metadata tables. CQL support for describing keyspaces / column familes - Key: CASSANDRA-2477 URL: https://issues.apache.org/jira/browse/CASSANDRA-2477 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Eric Evans Labels: cql Fix For: 0.8.1 Attachments: 2477-virtual-cfs-false-start.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050016#comment-13050016 ] Yang Yang commented on CASSANDRA-2774: -- ---Once compaction has compacted the deletes, all node will reach common agreement. we probably need to look at this more closely. I was going to use the example of a node always changing its report due to the different merging order in each read, but since you pointed out that we should allow all compaction to finish before making the judgement, let's look at another ill-case: the way I contrive the following case is very similar to the often-mentioned compaction ill-case, and we move that effect onto network messages, because changing the order of merging sstables in compaction is very similar to changing the order of message deliveries. let's say we have 4 nodes, A B C D. all the traffic we observe is, with increasing timestamp(): A leader add 1 ts=100 B leader delete ts=200 C leader add 2 ts=300 now the updates so far start to replicate to D: assume that D sees the following order: A.(add 1), C.(add 2), B.(delete), after these, D's state is: [A:1 C:2, last_delete=200, timestamp=300] now let's all the traffic between A,B,C go through, and they fully resolve (receiving pair-wise messages and etc), so A B C all come to state: [A:nil C:2, last_delete=200 timestamp=300] now A's state and D's state are different, let's say we let A repair D, A's A-shard has a lower clock, so D will win; if we let D repair A, A's A-shard is isDelta(), so it trumps D. as a result we never reach agreement between A and D, even though traffic is allowed to flow freely. I just started looking inside the CounterContext logic, so I could very well be wrong. Thanks for your time looking through this. as to performance, I don't think it will be a significant increase: 1) most application use cases will increment the same counter for many times, while it is in memtable. it's hard to imagine that most counters will be incremented only once before being dumped out to sstable. only the first increment in memtable for each counter will suffer a read into disk, if on average each counter is incremented 1000 times before being flushed, the disk read cost is amortized over 1000 increments; 2) even if we do the disk read, any realistic counter setup already needs ROW and CLONE, so a disk read is needed anyway before the client is acked. here we do an extra disk read, but when we do the regular disk read for CounterMutation.makeReplicationMutation() , the file blocks are already brought into cache by the new extra read, so it saves time, and total disk access time remains roughly the same. one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2774) one way to make counter delete work better
[ https://issues.apache.org/jira/browse/CASSANDRA-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050025#comment-13050025 ] Yang Yang commented on CASSANDRA-2774: -- one further thing: even if the current implementation reaches a common agreement, it can be some very arbitrary state:yes everyone uses the same rules so if messages are all played out, there is got to be the same final winner. but the winner wins by some very *arbitrary* rules---you can't reason about the final state. in the new implementation, the end result is a serializable state, meaning that there exists a sequence of a subset of incoming messages, so that if they are played out on a node in serial, we would achieve the final result, then the final result can be explained to users to be due to such a sequence of messages, after message loss and re-ordering. I don't think the current implementation has this property one way to make counter delete work better -- Key: CASSANDRA-2774 URL: https://issues.apache.org/jira/browse/CASSANDRA-2774 Project: Cassandra Issue Type: New Feature Affects Versions: 0.8.0 Reporter: Yang Yang Attachments: counter_delete.diff current Counter does not work with delete, because different merging order of sstables would produces different result, for example: add 1 delete add 2 if the merging happens by 1-2, (1,2)--3 order, the result we see will be 2 if merging is: 1--3, (1,3)--2, the result will be 3. the issue is that delete now can not separate out previous adds and adds later than the delete. supposedly a delete is to create a completely new incarnation of the counter, or a new lifetime, or epoch. the new approach utilizes the concept of epoch number, so that each delete bumps up the epoch number. since each write is replicated (replicate on write is almost always enabled in practice, if this is a concern, we could further force ROW in case of delete ), so the epoch number is global to a replica set changes are attached, existing tests pass fine, some tests are modified since the semantic is changed a bit. some cql tests do not pass in the original 0.8.0 source, that's not the fault of this change. see details at http://mail-archives.apache.org/mod_mbox/cassandra-user/201106.mbox/%3cbanlktikqcglsnwtt-9hvqpseoo7sf58...@mail.gmail.com%3E the goal of this is to make delete work ( at least with consistent behavior, yes in case of long network partition, the behavior is not ideal, but it's consistent with the definition of logical clock), so that we could have expiring Counters -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2776) add ability to return endpoints to nodetool
add ability to return endpoints to nodetool
--
Key: CASSANDRA-2776
URL: https://issues.apache.org/jira/browse/CASSANDRA-2776
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
Fix For: 0.8.1
- add nodetool endpoints
- take care of this FIXME:
{code}
public ListInetAddress getEndpoints(String keyspace, String key)
{
// FIXME: string key
return ssProxy.getNaturalEndpoints(keyspace, key.getBytes(UTF_8));
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2776) add ability to return endpoints to nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patricio Echague updated CASSANDRA-2776:
Description:
- add nodetool endpoints
- take care of this FIXME:
{code}
public ListInetAddress getEndpoints(String keyspace, String key)
{
// FIXME: string key
return ssProxy.getNaturalEndpoints(keyspace, key.getBytes(UTF_8));
}
{code}
NOTE: Use the key type info we have now, to decode the key-as-string.
was:
- add nodetool endpoints
- take care of this FIXME:
{code}
public ListInetAddress getEndpoints(String keyspace, String key)
{
// FIXME: string key
return ssProxy.getNaturalEndpoints(keyspace, key.getBytes(UTF_8));
}
{code}
add ability to return endpoints to nodetool
--
Key: CASSANDRA-2776
URL: https://issues.apache.org/jira/browse/CASSANDRA-2776
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
Labels: nodetool, tools
Fix For: 0.8.1
- add nodetool endpoints
- take care of this FIXME:
{code}
public ListInetAddress getEndpoints(String keyspace, String key)
{
// FIXME: string key
return ssProxy.getNaturalEndpoints(keyspace, key.getBytes(UTF_8));
}
{code}
NOTE: Use the key type info we have now, to decode the key-as-string.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-1248) ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
[
https://issues.apache.org/jira/browse/CASSANDRA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis reassigned CASSANDRA-1248:
-
Assignee: Patricio Echague
Patricio, can you see if you can reproduce on a VM w/ a very small data
partition?
Easiest way to make sure Cassandra's is there enough space left checks don't
get in the way is probably to comment them out.
ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
Key: CASSANDRA-1248
URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 0.6.3
java version 1.6.0_0
IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
Ubuntu 8.10
6 node cluster
RF==3
Reporter: David King
Assignee: Patricio Echague
Fix For: 0.8.2
A node died by running out of disk space. After growing the volume, trying to
execute reads dies with:
{quote}
ERROR 11:02:51,957 Uncaught exception in thread
Thread[ROW-READ-STAGE:10,5,main]
java.lang.ArrayIndexOutOfBoundsException
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
at
org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.init(SSTableSliceIterator.java:125)
at
org.apache.cassandra.db.filter.SSTableSliceIterator.init(SSTableSliceIterator.java:59)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
at
org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
at org.apache.cassandra.db.Table.getRow(Table.java:381)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{quote}
The clients then die with TimeoutExceptions.
It might be better to return an exception to the client immediately instead
of letting them die, and it would be helpful if this exception were
accompanied with a message indicating *which* SStable is malformed.
This command does *not* turn up the failed table (nor does the same with
sstablekeys):
{quote}
ri@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x;
/usr/local/cassandra/bin/sstable2json $x /dev/null; done
{quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1136218 - in /cassandra/drivers/py/cql: connection.py cursor.py
Author: jbellis
Date: Wed Jun 15 22:00:02 2011
New Revision: 1136218
URL: http://svn.apache.org/viewvc?rev=1136218view=rev
Log:
cleanup
Modified:
cassandra/drivers/py/cql/connection.py
cassandra/drivers/py/cql/cursor.py
Modified: cassandra/drivers/py/cql/connection.py
URL:
http://svn.apache.org/viewvc/cassandra/drivers/py/cql/connection.py?rev=1136218r1=1136217r2=1136218view=diff
==
--- cassandra/drivers/py/cql/connection.py (original)
+++ cassandra/drivers/py/cql/connection.py Wed Jun 15 22:00:02 2011
@@ -23,7 +23,6 @@ from cql.cassandra.ttypes import Authent
class Connection(object):
-
def __init__(self, host, port, keyspace, user=None, password=None):
Params:
Modified: cassandra/drivers/py/cql/cursor.py
URL:
http://svn.apache.org/viewvc/cassandra/drivers/py/cql/cursor.py?rev=1136218r1=1136217r2=1136218view=diff
==
--- cassandra/drivers/py/cql/cursor.py (original)
+++ cassandra/drivers/py/cql/cursor.py Wed Jun 15 22:00:02 2011
@@ -34,14 +34,13 @@ _COUNT_DESCRIPTION = (None, None, None,
_VOID_DESCRIPTION = (None)
class Cursor:
-
_keyspace_re = re.compile(USE (\w+);?, re.I | re.M)
_cfamily_re = re.compile(\s*SELECT\s+.+\s+FROM\s+[\']?(\w+), re.I | re.M)
_ddl_re = re.compile(\s*(CREATE|ALTER|DROP)\s+, re.I | re.M)
def __init__(self, parent_connection):
self.open_socket = True
-self.parent_connection = parent_connection
+self._connection = parent_connection
self.description = None # A list of 7-tuples:
# (column_name, type_code, none, none,
@@ -52,7 +51,7 @@ class Cursor:
self.rowcount = -1 # Populate on execute()
self.compression = 'GZIP'
-self._query_ks = self.parent_connection.keyspace
+self._query_ks = self._connection.keyspace
self._query_cf = None
self.decoder = SchemaDecoder(self.__get_schema())
@@ -103,7 +102,7 @@ class Cursor:
return d
schema = {}
-client = self.parent_connection.client
+client = self._connection.client
for ksdef in client.describe_keyspaces():
schema[ksdef.name] = column_families(ksdef.cf_defs)
return schema
@@ -126,7 +125,7 @@ class Cursor:
request_compression = getattr(Compression, self.compression)
try:
-client = self.parent_connection.client
+client = self._connection.client
response = client.execute_cql_query(compressed_q,
request_compression)
except InvalidRequestException, ire:
raise cql.ProgrammingError(Bad Request: %s % ire.why)
@@ -186,11 +185,11 @@ class Cursor:
row = self.result[self.rs_idx]
self.rs_idx += 1
-if self.description != _COUNT_DESCRIPTION:
+if self.description == _COUNT_DESCRIPTION:
+return row
+else:
self.description = self.decoder.decode_description(self._query_ks,
self._query_cf, row)
return self.decoder.decode_row(self._query_ks, self._query_cf, row)
-else:
-return row
def fetchmany(self, size=None):
self.__checksock()
@@ -249,4 +248,4 @@ class Cursor:
def __checksock(self):
if not self.open_socket:
raise cql.InternalError(Cursor belonging to %s has been closed. %
-(self.parent_connection, ))
+(self._connection, ))
[jira] [Resolved] (CASSANDRA-2471) txcql driver broken
[ https://issues.apache.org/jira/browse/CASSANDRA-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2471. --- Resolution: Later Fix Version/s: (was: 0.8.2) Assignee: (was: paul cannon) twisted driver has been broken for weeks (months?) and nobody has cared. I've removed it from the drivers/ tree for now. txcql driver broken --- Key: CASSANDRA-2471 URL: https://issues.apache.org/jira/browse/CASSANDRA-2471 Project: Cassandra Issue Type: Bug Components: API Reporter: Eric Evans Labels: cql The Plain Old Python driver was updated for dbapi2, breaking txcql. Please do the needful. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1974) PFEPS-like snitch that uses gossip instead of a property file
[ https://issues.apache.org/jira/browse/CASSANDRA-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-1974: -- Priority: Minor (was: Major) Affects Version/s: (was: 0.7.0) Fix Version/s: 0.8.2 PFEPS-like snitch that uses gossip instead of a property file - Key: CASSANDRA-1974 URL: https://issues.apache.org/jira/browse/CASSANDRA-1974 Project: Cassandra Issue Type: New Feature Reporter: Brandon Williams Assignee: Brandon Williams Priority: Minor Fix For: 0.8.2 Now that we have an ec2 snitch that propagates its rack/dc info via gossip from CASSANDRA-1654, it doesn't make a lot of sense to use PFEPS where you have to rsync the property file across all the machines when you add a node. Instead, we could have a snitch where you specify its rack/dc in a property file, and propagate this via gossip like the ec2 snitch. In order to not break PFEPS, this should probably be a new snitch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050081#comment-13050081 ] Alan Liang commented on CASSANDRA-2769: --- Instead of letting DataTracker#markCompacting modify the subset of sstables to be compacted, I think it might be cleaner if it didn't and relied on the CompactionStrategy to select the correct sstables. We can do this by having the CompactionStrategy get the non compacting sstables from the DataTracker and work with those to generate the buckets. The strategy should also be responsible for creating buckets that fit within the min/max thresholds. #markCompacting would then be changed such that it can either accept/reject a bucket to be compacted instead of modifying the subset. #markCompacting will also serve to handle the race condition of the DataTracker being inaccurate, whereby, it will move on to other buckets. With this, we can avoid generating buckets that are already compacting and it gives full control of what actually is compacted by the CompactionStrategy. What do you guys think? Cannot Create Duplicate Compaction Marker - Key: CASSANDRA-2769 URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.0 Reporter: Benjamin Coverston Assignee: Sylvain Lebresne Fix For: 0.8.2 Attachments: 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch, 0001-Do-compact-only-smallerSSTables.patch, 0002-Only-compact-what-has-been-succesfully-marked-as-com.patch Concurrent compaction can trigger the following exception when two threads compact the same sstable. DataTracker attempts to prevent this but apparently not successfully. java.io.IOError: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2777) Pig storage handler should implement LoadMetadata
Pig storage handler should implement LoadMetadata - Key: CASSANDRA-2777 URL: https://issues.apache.org/jira/browse/CASSANDRA-2777 Project: Cassandra Issue Type: Improvement Components: Contrib Reporter: Brandon Williams Assignee: Brandon Williams Priority: Minor The reason for this is many builtin functions like SUM won't work on longs (you can workaround using LongSum, but that's lame) because the query planner doesn't know about the types beforehand, even though we are casting to native longs. There is some impact to this, though. With LoadMetadata implemented, existing scripts that specify schema will need to remove it (since LM is doing it for them) and they will need to conform to LM's terminology (key, columns, name, value) within the script. This is trivial to change, however, and the increased functionality is worth the switch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2491) A new config parameter, broadcast_address
[ https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050101#comment-13050101 ] Brandon Williams commented on CASSANDRA-2491: - One last thing I overlooked earlier is the snitches have getBCA(). This seems unnecessary since anything that needs it can access DD, which is all the snitches are doing. This changes the snitch interface and breaks any custom snitches. A new config parameter, broadcast_address - Key: CASSANDRA-2491 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491 Project: Cassandra Issue Type: Improvement Components: Core Environment: x86_64 GNU/Linux Reporter: Khee Chin Assignee: Khee Chin Priority: Trivial Labels: patch Fix For: 1.0 Attachments: 2491_broadcast_address.patch, 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch, 2491_broadcast_address_v4.patch Original Estimate: 336h Remaining Estimate: 336h A new config parameter, broadcast_address In a cluster setup where one or more nodes is behind a firewall and has a private ip address, listen_address does not allow the hosts behind the firewalls to be discovered by other nodes. Attached is a patch that introduces a new config parameter broadcast_address which allows Cassandra nodes to explicitly specify their external ip address. In addition, this allows listen_address to be set to 0.0.0.0 on the already firewalled node. broadcast_address fallsback to listen_address when it is not stated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2777) Pig storage handler should implement LoadMetadata
[ https://issues.apache.org/jira/browse/CASSANDRA-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-2777: Reviewer: jeromatron Pig storage handler should implement LoadMetadata - Key: CASSANDRA-2777 URL: https://issues.apache.org/jira/browse/CASSANDRA-2777 Project: Cassandra Issue Type: Improvement Components: Contrib Reporter: Brandon Williams Assignee: Brandon Williams Priority: Minor Attachments: 2777.txt The reason for this is many builtin functions like SUM won't work on longs (you can workaround using LongSum, but that's lame) because the query planner doesn't know about the types beforehand, even though we are casting to native longs. There is some impact to this, though. With LoadMetadata implemented, existing scripts that specify schema will need to remove it (since LM is doing it for them) and they will need to conform to LM's terminology (key, columns, name, value) within the script. This is trivial to change, however, and the increased functionality is worth the switch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2777) Pig storage handler should implement LoadMetadata
[ https://issues.apache.org/jira/browse/CASSANDRA-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-2777: Fix Version/s: 0.7.7 Pig storage handler should implement LoadMetadata - Key: CASSANDRA-2777 URL: https://issues.apache.org/jira/browse/CASSANDRA-2777 Project: Cassandra Issue Type: Improvement Components: Contrib Reporter: Brandon Williams Assignee: Brandon Williams Priority: Minor Fix For: 0.7.7 Attachments: 2777.txt The reason for this is many builtin functions like SUM won't work on longs (you can workaround using LongSum, but that's lame) because the query planner doesn't know about the types beforehand, even though we are casting to native longs. There is some impact to this, though. With LoadMetadata implemented, existing scripts that specify schema will need to remove it (since LM is doing it for them) and they will need to conform to LM's terminology (key, columns, name, value) within the script. This is trivial to change, however, and the increased functionality is worth the switch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2614) create Column and CounterColumn in the same column family
[ https://issues.apache.org/jira/browse/CASSANDRA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050153#comment-13050153 ] Dave Rav commented on CASSANDRA-2614: - Is it possible to loop through all the columns that need to be deleted find there type (regular or counter) and delete them base on there type? create Column and CounterColumn in the same column family - Key: CASSANDRA-2614 URL: https://issues.apache.org/jira/browse/CASSANDRA-2614 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Dave Rav Assignee: Sylvain Lebresne Priority: Minor Fix For: 0.8.2 create Column and CounterColumn in the same column family -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2778) Unable to set compaction strategy in cli using create column family command
Unable to set compaction strategy in cli using create column family command
---
Key: CASSANDRA-2778
URL: https://issues.apache.org/jira/browse/CASSANDRA-2778
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
The following command does not set compaction strategy and its options:
{code}
create column family Standard1
with comparator = BytesType
and compaction_strategy =
'org.apache.cassandra.db.compaction.TimestampBucketedCompactionStrategy'
and compaction_strategy_options = [{max_sstable_size:504857600,
retention_in_seconds:60}];
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2777) Pig storage handler should implement LoadMetadata
[
https://issues.apache.org/jira/browse/CASSANDRA-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050159#comment-13050159
]
Jeremy Hanna edited comment on CASSANDRA-2777 at 6/16/11 12:21 AM:
---
while we're add it can we remove the redundant addMutation call on line 505 and
on line 513 add the e param on:
{code}
throw new IOException(e + Output must be (key, {(column,value)...}) for
ColumnFamily or (key, {supercolumn:{(column,value)...}...}) for
SuperColumnFamily, e);
{code}
was (Author: jeromatron):
while we're add it can we remove the redundant addMutation call on line 505
and on line 513 add the e param on:
{quote}
throw new IOException(e + Output must be (key, {(column,value)...}) for
ColumnFamily or (key, {supercolumn:{(column,value)...}...}) for
SuperColumnFamily, e);
{quote}
Pig storage handler should implement LoadMetadata
-
Key: CASSANDRA-2777
URL: https://issues.apache.org/jira/browse/CASSANDRA-2777
Project: Cassandra
Issue Type: Improvement
Components: Contrib
Reporter: Brandon Williams
Assignee: Brandon Williams
Priority: Minor
Fix For: 0.7.7
Attachments: 2777.txt
The reason for this is many builtin functions like SUM won't work on longs
(you can workaround using LongSum, but that's lame) because the query planner
doesn't know about the types beforehand, even though we are casting to native
longs.
There is some impact to this, though. With LoadMetadata implemented,
existing scripts that specify schema will need to remove it (since LM is
doing it for them) and they will need to conform to LM's terminology (key,
columns, name, value) within the script. This is trivial to change, however,
and the increased functionality is worth the switch.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2777) Pig storage handler should implement LoadMetadata
[
https://issues.apache.org/jira/browse/CASSANDRA-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050159#comment-13050159
]
Jeremy Hanna commented on CASSANDRA-2777:
-
while we're add it can we remove the redundant addMutation call on line 505 and
on line 513 add the e param on:
{quote}
throw new IOException(e + Output must be (key, {(column,value)...}) for
ColumnFamily or (key, {supercolumn:{(column,value)...}...}) for
SuperColumnFamily, e);
{quote}
Pig storage handler should implement LoadMetadata
-
Key: CASSANDRA-2777
URL: https://issues.apache.org/jira/browse/CASSANDRA-2777
Project: Cassandra
Issue Type: Improvement
Components: Contrib
Reporter: Brandon Williams
Assignee: Brandon Williams
Priority: Minor
Fix For: 0.7.7
Attachments: 2777.txt
The reason for this is many builtin functions like SUM won't work on longs
(you can workaround using LongSum, but that's lame) because the query planner
doesn't know about the types beforehand, even though we are casting to native
longs.
There is some impact to this, though. With LoadMetadata implemented,
existing scripts that specify schema will need to remove it (since LM is
doing it for them) and they will need to conform to LM's terminology (key,
columns, name, value) within the script. This is trivial to change, however,
and the increased functionality is worth the switch.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2777) Pig storage handler should implement LoadMetadata
[ https://issues.apache.org/jira/browse/CASSANDRA-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050160#comment-13050160 ] Brandon Williams commented on CASSANDRA-2777: - Sure, will add those on commit. Pig storage handler should implement LoadMetadata - Key: CASSANDRA-2777 URL: https://issues.apache.org/jira/browse/CASSANDRA-2777 Project: Cassandra Issue Type: Improvement Components: Contrib Reporter: Brandon Williams Assignee: Brandon Williams Priority: Minor Fix For: 0.7.7 Attachments: 2777.txt The reason for this is many builtin functions like SUM won't work on longs (you can workaround using LongSum, but that's lame) because the query planner doesn't know about the types beforehand, even though we are casting to native longs. There is some impact to this, though. With LoadMetadata implemented, existing scripts that specify schema will need to remove it (since LM is doing it for them) and they will need to conform to LM's terminology (key, columns, name, value) within the script. This is trivial to change, however, and the increased functionality is worth the switch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2491) A new config parameter, broadcast_address
[ https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-2491: - Attachment: 2491_broadcast_address_v5.patch Sure... will try that in another ticket... Thanks... A new config parameter, broadcast_address - Key: CASSANDRA-2491 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491 Project: Cassandra Issue Type: Improvement Components: Core Environment: x86_64 GNU/Linux Reporter: Khee Chin Assignee: Khee Chin Priority: Trivial Labels: patch Fix For: 1.0 Attachments: 2491_broadcast_address.patch, 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch, 2491_broadcast_address_v4.patch, 2491_broadcast_address_v5.patch Original Estimate: 336h Remaining Estimate: 336h A new config parameter, broadcast_address In a cluster setup where one or more nodes is behind a firewall and has a private ip address, listen_address does not allow the hosts behind the firewalls to be discovered by other nodes. Attached is a patch that introduces a new config parameter broadcast_address which allows Cassandra nodes to explicitly specify their external ip address. In addition, this allows listen_address to be set to 0.0.0.0 on the already firewalled node. broadcast_address fallsback to listen_address when it is not stated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2614) create Column and CounterColumn in the same column family
[ https://issues.apache.org/jira/browse/CASSANDRA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050153#comment-13050153 ] Dave Rav edited comment on CASSANDRA-2614 at 6/16/11 2:48 AM: -- Is it possible to loop through all the columns that need to be deleted find there type, 'validation_class' (regular or CounterColumnType) and delete them base on there type? was (Author: daver...@yahoo.com): Is it possible to loop through all the columns that need to be deleted find there type (regular or counter) and delete them base on there type? create Column and CounterColumn in the same column family - Key: CASSANDRA-2614 URL: https://issues.apache.org/jira/browse/CASSANDRA-2614 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Dave Rav Assignee: Sylvain Lebresne Priority: Minor Fix For: 0.8.2 create Column and CounterColumn in the same column family -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2779) files not cleaned up by GC?
files not cleaned up by GC? --- Key: CASSANDRA-2779 URL: https://issues.apache.org/jira/browse/CASSANDRA-2779 Project: Cassandra Issue Type: Bug Reporter: Terje Marthinussen Priority: Critical Fix For: 0.8.0 This is 0.8.0 + a few 0.8.1 patches on repair. We tested repair on 2 nodes in the cluster last night. Interestingly enough, I don't believe the node described here is in any way neighbour of the nodes we tested repair on so I am not sure why it is streaming data both in and out, but in any case, it has joined the streaming party. We now see: ERROR [CompactionExecutor:5] 2011-06-16 09:12:23,928 CompactionManager.java (line 510) insufficient space to compact even the two smallest files, aborting INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space And we see a lot of them: INFO [CompactionExecutor:5] 2011-06-16 09:11:59,164 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:46,489 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:17:53,299 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:17,782 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:42,078 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:06,984 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:32,079 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:57,265 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:22,706 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:47,331 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:13,062 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:38,288 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:03,500 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:29,407 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:55,577 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:20,951 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:46,448 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:24:12,030 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:00,633 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:26,119 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:49,002 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:20,196 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:45,322 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:11:07,619 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:01:45,562 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:02:10,236 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:05:31,297 StorageService.java (line 2071) requesting GC to free disk space Available disk is 105GB and it is trying to compact a set of the largest sstables. There is probably easily enough disk to do so, but the estimation is not sufficient (lots of dupes here after streaming I think, probably heavily affected by CASSANDRA-2698). It is trying to compact 2 sstables of 58 and 41GB. If I look at the data dir, I see 46 *Compacted files which makes up an additional 137GB of space. The oldest of these Compacted files dates back to Jun 16th 01:26, so 10 hours old. It does however succeed at cleaning up some files. There are definitely files which do get deleted. Just that there is a lot which is not. Either the GC cleanup tactic is seriously
[jira] [Commented] (CASSANDRA-2779) files not cleaned up by GC?
[ https://issues.apache.org/jira/browse/CASSANDRA-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050191#comment-13050191 ] Terje Marthinussen commented on CASSANDRA-2779: --- Interestingly enough... Just as I posted this it started removing the files and the case study disappeared and started working fine. A few minutes before doing so, if completed manual repair, so yes, I guess repair is a likely candidate for keeping file references. If I remember right, it builds a list of files to stream at the start. They may not be dereferenced fully until the streaming finishes? files not cleaned up by GC? --- Key: CASSANDRA-2779 URL: https://issues.apache.org/jira/browse/CASSANDRA-2779 Project: Cassandra Issue Type: Bug Reporter: Terje Marthinussen Priority: Critical Fix For: 0.8.0 This is 0.8.0 + a few 0.8.1 patches on repair. We tested repair on 2 nodes in the cluster last night. Interestingly enough, I don't believe the node described here is in any way neighbour of the nodes we tested repair on so I am not sure why it is streaming data both in and out, but in any case, it has joined the streaming party. We now see: ERROR [CompactionExecutor:5] 2011-06-16 09:12:23,928 CompactionManager.java (line 510) insufficient space to compact even the two smallest files, aborting INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space And we see a lot of them: INFO [CompactionExecutor:5] 2011-06-16 09:11:59,164 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:46,489 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:17:53,299 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:17,782 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:42,078 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:06,984 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:32,079 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:57,265 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:22,706 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:47,331 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:13,062 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:38,288 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:03,500 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:29,407 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:55,577 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:20,951 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:46,448 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:24:12,030 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:00,633 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:26,119 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:49,002 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:20,196 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:45,322 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:11:07,619 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:01:45,562 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:02:10,236 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:05:31,297
[jira] [Updated] (CASSANDRA-2779) files not cleaned up by GC?
[ https://issues.apache.org/jira/browse/CASSANDRA-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terje Marthinussen updated CASSANDRA-2779: -- Priority: Major (was: Critical) Dropping to major again. Things do get cleaned up, eventually... files not cleaned up by GC? --- Key: CASSANDRA-2779 URL: https://issues.apache.org/jira/browse/CASSANDRA-2779 Project: Cassandra Issue Type: Bug Reporter: Terje Marthinussen Fix For: 0.8.0 This is 0.8.0 + a few 0.8.1 patches on repair. We tested repair on 2 nodes in the cluster last night. Interestingly enough, I don't believe the node described here is in any way neighbour of the nodes we tested repair on so I am not sure why it is streaming data both in and out, but in any case, it has joined the streaming party. We now see: ERROR [CompactionExecutor:5] 2011-06-16 09:12:23,928 CompactionManager.java (line 510) insufficient space to compact even the two smallest files, aborting INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space And we see a lot of them: INFO [CompactionExecutor:5] 2011-06-16 09:11:59,164 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:46,489 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:17:53,299 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:17,782 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:42,078 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:06,984 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:32,079 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:57,265 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:22,706 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:47,331 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:13,062 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:38,288 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:03,500 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:29,407 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:55,577 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:20,951 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:46,448 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:24:12,030 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:00,633 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:26,119 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:49,002 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:20,196 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:45,322 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:11:07,619 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:01:45,562 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:02:10,236 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:05:31,297 StorageService.java (line 2071) requesting GC to free disk space Available disk is 105GB and it is trying to compact a set of the largest sstables. There is probably easily enough disk to do so, but the estimation is not sufficient (lots of dupes here after streaming I think, probably heavily affected by CASSANDRA-2698). It is trying to compact 2 sstables of 58 and 41GB. If I
[jira] [Commented] (CASSANDRA-2521) Move away from Phantom References for Compaction/Memtable
[ https://issues.apache.org/jira/browse/CASSANDRA-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050203#comment-13050203 ] Jeff Kesselman commented on CASSANDRA-2521: --- Relying on either finalizers or phantom references for your own clean up is not correct. These mechanisms are not triggered when an object becomes a candidate for collection, but only when the space is actually reclaimed. The time between these two states is up to the gc implementation and can be infinite in that the gc is never *required* to collect any given object unless failure to collect that object would result in an Out Of Memory condition. GCs often only partially collect, choosing the easiest objects to clean up first and going no further if no more space is required. A null gc that never collects space is actually perfectly legal under the java VM specification and has actually been used by IBM in the past for certain operational environments. Move away from Phantom References for Compaction/Memtable - Key: CASSANDRA-2521 URL: https://issues.apache.org/jira/browse/CASSANDRA-2521 Project: Cassandra Issue Type: Improvement Reporter: Chris Goffinet Assignee: Sylvain Lebresne Fix For: 1.0 http://wiki.apache.org/cassandra/MemtableSSTable Let's move to using reference counting instead of relying on GC to be called in StorageService. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2521) Move away from Phantom References for Compaction/Memtable
[ https://issues.apache.org/jira/browse/CASSANDRA-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050204#comment-13050204 ] Jeff Kesselman commented on CASSANDRA-2521: --- It should also be noted that any object with either a finalizer or a phantom reference to it cannot be cleaned up in the eden by the hotspot generational collector and is forced to proceed to the next generation where gc is significantly more expensive. Whether this has any real effect on your gc times depends on the lifetime of the object in question. Creating a lot of short lived objects with finalizers or phantom references is bad. Creating middle to long lived objects with finalizers or phantom references is less of a problem. Move away from Phantom References for Compaction/Memtable - Key: CASSANDRA-2521 URL: https://issues.apache.org/jira/browse/CASSANDRA-2521 Project: Cassandra Issue Type: Improvement Reporter: Chris Goffinet Assignee: Sylvain Lebresne Fix For: 1.0 http://wiki.apache.org/cassandra/MemtableSSTable Let's move to using reference counting instead of relying on GC to be called in StorageService. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2762) Token cannot contain comma (possibly non-alpha/non-numeric too?) in OrderPreservingPartitioner
[ https://issues.apache.org/jira/browse/CASSANDRA-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050206#comment-13050206 ] Jonathan Ellis commented on CASSANDRA-2762: --- However we should make sure that we fail-fast invalid initial_token setting, as well as do not hand out bad tokens during auto bootstrap token selection Token cannot contain comma (possibly non-alpha/non-numeric too?) in OrderPreservingPartitioner -- Key: CASSANDRA-2762 URL: https://issues.apache.org/jira/browse/CASSANDRA-2762 Project: Cassandra Issue Type: Bug Reporter: Jackson Chung It'd appear that when the token contain comma in the OrderPreservingPartitioner case, C* will fail with assert error: ERROR [GossipStage:1] 2011-06-09 16:01:05,063 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[GossipStage:1,5,main] java.lang.AssertionError at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:685) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772) at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679) at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2762) Token cannot contain comma (possibly non-alpha/non-numeric too?) in OrderPreservingPartitioner
[ https://issues.apache.org/jira/browse/CASSANDRA-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reopened CASSANDRA-2762: --- Assignee: Jonathan Ellis Token cannot contain comma (possibly non-alpha/non-numeric too?) in OrderPreservingPartitioner -- Key: CASSANDRA-2762 URL: https://issues.apache.org/jira/browse/CASSANDRA-2762 Project: Cassandra Issue Type: Bug Reporter: Jackson Chung Assignee: Jonathan Ellis It'd appear that when the token contain comma in the OrderPreservingPartitioner case, C* will fail with assert error: ERROR [GossipStage:1] 2011-06-09 16:01:05,063 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[GossipStage:1,5,main] java.lang.AssertionError at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:685) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:648) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:772) at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:737) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:679) at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:60) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2779) files not cleaned up by GC?
[ https://issues.apache.org/jira/browse/CASSANDRA-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2779. --- Resolution: Not A Problem Fix Version/s: (was: 0.8.0) I don't see a problem here (I think repair holding references to files being streamed is totally reasonable) files not cleaned up by GC? --- Key: CASSANDRA-2779 URL: https://issues.apache.org/jira/browse/CASSANDRA-2779 Project: Cassandra Issue Type: Bug Reporter: Terje Marthinussen This is 0.8.0 + a few 0.8.1 patches on repair. We tested repair on 2 nodes in the cluster last night. Interestingly enough, I don't believe the node described here is in any way neighbour of the nodes we tested repair on so I am not sure why it is streaming data both in and out, but in any case, it has joined the streaming party. We now see: ERROR [CompactionExecutor:5] 2011-06-16 09:12:23,928 CompactionManager.java (line 510) insufficient space to compact even the two smallest files, aborting INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space And we see a lot of them: INFO [CompactionExecutor:5] 2011-06-16 09:11:59,164 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:46,489 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:17:53,299 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:17,782 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:42,078 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:06,984 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:32,079 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:57,265 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:22,706 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:47,331 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:13,062 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:38,288 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:03,500 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:29,407 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:55,577 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:20,951 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:46,448 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:24:12,030 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:00,633 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:26,119 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:49,002 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:20,196 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:45,322 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:11:07,619 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:01:45,562 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:02:10,236 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:05:31,297 StorageService.java (line 2071) requesting GC to free disk space Available disk is 105GB and it is trying to compact a set of the largest sstables. There is probably easily enough disk to do so, but the estimation is not sufficient (lots of dupes here after streaming I think, probably heavily affected by CASSANDRA-2698). It is trying to
svn commit: r1136287 - in /cassandra/trunk: ./ src/java/org/apache/cassandra/db/ src/java/org/apache/cassandra/db/columniterator/ src/java/org/apache/cassandra/db/compaction/ src/java/org/apache/cassa
Author: jbellis Date: Thu Jun 16 04:24:33 2011 New Revision: 1136287 URL: http://svn.apache.org/viewvc?rev=1136287view=rev Log: replace CollatingIterator, ReducingIterator with MergeIterator patch by stuhood; reviewed by jbellis for CASSANDRA-2062 Added: cassandra/trunk/test/unit/org/apache/cassandra/utils/MergeIteratorTest.java Removed: cassandra/trunk/src/java/org/apache/cassandra/db/RowIterator.java cassandra/trunk/src/java/org/apache/cassandra/utils/ReducingIterator.java Modified: cassandra/trunk/CHANGES.txt cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java cassandra/trunk/src/java/org/apache/cassandra/db/RowIteratorFactory.java cassandra/trunk/src/java/org/apache/cassandra/db/columniterator/IColumnIterator.java cassandra/trunk/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java cassandra/trunk/src/java/org/apache/cassandra/db/compaction/CompactionManager.java cassandra/trunk/src/java/org/apache/cassandra/db/compaction/LazilyCompactedRow.java cassandra/trunk/src/java/org/apache/cassandra/db/filter/QueryFilter.java cassandra/trunk/src/java/org/apache/cassandra/io/sstable/KeyIterator.java cassandra/trunk/src/java/org/apache/cassandra/io/sstable/ReducingKeyIterator.java cassandra/trunk/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java cassandra/trunk/src/java/org/apache/cassandra/service/RangeSliceResponseResolver.java cassandra/trunk/src/java/org/apache/cassandra/service/RowRepairResolver.java cassandra/trunk/src/java/org/apache/cassandra/utils/FBUtilities.java Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1136287r1=1136286r2=1136287view=diff == --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Thu Jun 16 04:24:33 2011 @@ -3,6 +3,8 @@ * add commitlog_total_space_in_mb to prevent fragmented logs (CASSANDRA-2427) * removed commitlog_rotation_threshold_in_mb configuration (CASSANDRA-2771) * make AbstractBounds.normalize de-overlapp overlapping ranges (CASSANDRA-2641) + * replace CollatingIterator, ReducingIterator with MergeIterator + (CASSANDRA-2062) 0.8.1 Modified: cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java?rev=1136287r1=1136286r2=1136287view=diff == --- cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java Thu Jun 16 04:24:33 2011 @@ -32,7 +32,6 @@ import javax.management.MBeanServer; import javax.management.ObjectName; import com.google.common.collect.Iterables; -import org.apache.commons.collections.IteratorUtils; import org.apache.commons.lang.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -1188,7 +1187,7 @@ public class ColumnFamilyStore implement IColumnIterator ci = filter.getMemtableColumnIterator(cached, null, getComparator()); ColumnFamily cf = ci.getColumnFamily().cloneMeShallow(); -filter.collectCollatedColumns(cf, ci, gcBefore); +filter.collateColumns(cf, Collections.singletonList(ci), getComparator(), gcBefore); // TODO this is necessary because when we collate supercolumns together, we don't check // their subcolumns for relevance, so we need to do a second prune post facto here. return cf.isSuper() ? removeDeleted(cf, gcBefore) : removeDeletedCF(cf, gcBefore); @@ -1244,10 +1243,7 @@ public class ColumnFamilyStore implement if (iterators.size() == 0) return null; -ComparatorIColumn comparator = filter.filter.getColumnComparator(getComparator()); -Iterator collated = IteratorUtils.collatedIterator(comparator, iterators); - -filter.collectCollatedColumns(returnCF, collated, gcBefore); +filter.collateColumns(returnCF, iterators, getComparator(), gcBefore); // Caller is responsible for final removeDeletedCF. This is important for cacheRow to work correctly: return returnCF; @@ -1298,7 +1294,7 @@ public class ColumnFamilyStore implement // It is fine to aliases the View.sstables since it's an unmodifiable collection CollectionSSTableReader sstables = currentView.sstables; -RowIterator iterator = RowIteratorFactory.getIterator(memtables, sstables, startWith, stopAt, filter, getComparator(), this); +CloseableIteratorRow iterator = RowIteratorFactory.getIterator(memtables, sstables, startWith, stopAt, filter, getComparator(), this); ListRow rows = new ArrayListRow(); try @@ -1486,7 +1482,7 @@ public class
[jira] [Commented] (CASSANDRA-2779) files not cleaned up by GC?
[ https://issues.apache.org/jira/browse/CASSANDRA-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050213#comment-13050213 ] Terje Marthinussen commented on CASSANDRA-2779: --- The problem is the disk use. I have no problem with the disk use if all recommendations for Cassandra is changed from typically 2x disk space required to hold the dataset to 4x disk space vs. what is needed to hold the dataset. Either we have to fix this issue to keep disk space down or we have to change the recommendations. files not cleaned up by GC? --- Key: CASSANDRA-2779 URL: https://issues.apache.org/jira/browse/CASSANDRA-2779 Project: Cassandra Issue Type: Bug Reporter: Terje Marthinussen This is 0.8.0 + a few 0.8.1 patches on repair. We tested repair on 2 nodes in the cluster last night. Interestingly enough, I don't believe the node described here is in any way neighbour of the nodes we tested repair on so I am not sure why it is streaming data both in and out, but in any case, it has joined the streaming party. We now see: ERROR [CompactionExecutor:5] 2011-06-16 09:12:23,928 CompactionManager.java (line 510) insufficient space to compact even the two smallest files, aborting INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space And we see a lot of them: INFO [CompactionExecutor:5] 2011-06-16 09:11:59,164 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:46,489 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:17:53,299 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:17,782 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:42,078 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:06,984 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:32,079 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:57,265 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:22,706 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:47,331 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:13,062 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:38,288 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:03,500 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:29,407 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:55,577 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:20,951 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:46,448 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:24:12,030 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:00,633 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:26,119 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:49,002 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:20,196 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:45,322 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:11:07,619 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:01:45,562 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:02:10,236 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:05:31,297 StorageService.java (line 2071) requesting GC to free disk space Available disk is 105GB and it is trying to compact a set of the
[jira] [Assigned] (CASSANDRA-2521) Move away from Phantom References for Compaction/Memtable
[ https://issues.apache.org/jira/browse/CASSANDRA-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reassigned CASSANDRA-2521: - Assignee: (was: Sylvain Lebresne) Sylvain has the largest backlog of any contributor, so I'm going to take this off his plate so someone with more time (Jeffrey?) can take a stab at it. Move away from Phantom References for Compaction/Memtable - Key: CASSANDRA-2521 URL: https://issues.apache.org/jira/browse/CASSANDRA-2521 Project: Cassandra Issue Type: Improvement Reporter: Chris Goffinet Fix For: 1.0 http://wiki.apache.org/cassandra/MemtableSSTable Let's move to using reference counting instead of relying on GC to be called in StorageService. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2779) files not cleaned up by GC?
[ https://issues.apache.org/jira/browse/CASSANDRA-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terje Marthinussen reopened CASSANDRA-2779: --- Reopening until a decision is made to fix either documentation or disk usage, Things like: http://wiki.apache.org/cassandra/CassandraHardware As covered in MemtableSSTable, compactions can require up to 100% of your in-use space temporarily in the worst case should not exist, and in addition to an up to 100% increase of data from streaming, you need 100% for compaction and headroom to avoid full GC from running every few minutes like occured in this Jira (basically an unusable service) files not cleaned up by GC? --- Key: CASSANDRA-2779 URL: https://issues.apache.org/jira/browse/CASSANDRA-2779 Project: Cassandra Issue Type: Bug Reporter: Terje Marthinussen This is 0.8.0 + a few 0.8.1 patches on repair. We tested repair on 2 nodes in the cluster last night. Interestingly enough, I don't believe the node described here is in any way neighbour of the nodes we tested repair on so I am not sure why it is streaming data both in and out, but in any case, it has joined the streaming party. We now see: ERROR [CompactionExecutor:5] 2011-06-16 09:12:23,928 CompactionManager.java (line 510) insufficient space to compact even the two smallest files, aborting INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space And we see a lot of them: INFO [CompactionExecutor:5] 2011-06-16 09:11:59,164 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:23,929 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:5] 2011-06-16 09:12:46,489 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:17:53,299 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:17,782 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:18:42,078 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:06,984 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:32,079 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:19:57,265 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:22,706 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:20:47,331 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:13,062 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:21:38,288 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:03,500 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:29,407 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:22:55,577 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:20,951 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:23:46,448 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:3] 2011-06-16 09:24:12,030 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:00,633 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:26,119 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 09:48:49,002 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:20,196 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:10:45,322 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:6] 2011-06-16 10:11:07,619 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:01:45,562 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:02:10,236 StorageService.java (line 2071) requesting GC to free disk space INFO [CompactionExecutor:7] 2011-06-16 11:05:31,297 StorageService.java
