date:20110513


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032871#comment-13032871
 ] 

Sylvain Lebresne commented on CASSANDRA-2644:
-

Not fully related to the discussion here but streaming is another part of 
bootstrap so I'll mention that CASSANDRA-2433 introduces some mechanism to 
handle unrecoverable failures during streaming (that is, streaming already 
retry on errors but 1) it retries indefinitely while the CASSANDRA-2433 
introduce a max retry and 2) it doesn't detect the other end being dead). 
Anyway, just referencing the ticket so that if this ticket becomes make 
bootstrap handle failures better, we don't duplicate efforts.   

 Make bootstrap retry
 

 Key: CASSANDRA-2644
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2644
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0 beta 2
Reporter: Chris Goffinet
Assignee: Chris Goffinet
 Fix For: 0.8.1

 Attachments: 
 0001-Make-ExpiringMap-have-objects-with-specific-timeouts.patch, 
 0002-Make-bootstrap-retry-and-increment-timeout-for-every.patch


 We ran into a situation where we had rpc_timeout set to 1 second, and the 
 node needing to compute the token took over a second (1.6 seconds). The 
 bootstrapping node hangs forever without getting a token because the expiring 
 map removes it before the reply comes back.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1102594 - /cassandra/branches/cassandra-0.7/CHANGES.txt

2011-05-13 Thread slebresne

Author: slebresne
Date: Fri May 13 07:58:27 2011
New Revision: 1102594

URL: http://svn.apache.org/viewvc?rev=1102594view=rev
Log:
Fix changelog

Modified:
cassandra/branches/cassandra-0.7/CHANGES.txt

Modified: cassandra/branches/cassandra-0.7/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1102594r1=1102593r2=1102594view=diff
==
--- cassandra/branches/cassandra-0.7/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.7/CHANGES.txt Fri May 13 07:58:27 2011
@@ -26,6 +26,8 @@
  * add quote-escaping via backslash to CLI (CASSANDRA-2623)
  * fig pig example script (CASSANDRA-2487)
  * fix dynamic snitch race in adding latencies (CASSANDRA-2618)
+ * Start/stop cassandra after more important services such as mdadm in
+   debian packaging (CASSANDRA-2481)
 
 
 0.7.5

[jira] [Commented] (CASSANDRA-2643) read repair/reconciliation breaks slice based iteration at QUORUM

2011-05-13 Thread Peter Schuller (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13032897#comment-13032897
 ] 

Peter Schuller commented on CASSANDRA-2643:
---

I realized I failed to make the possibly most important point: That you can 
indeed get short reads such that iteration will stop early. Consider 3 nodes, 
RF=3, QUORUM.

* Node A has [10,40] of columns
* Node B has [10,40] of columns
* Node C has [10,20] of column deletions for the columns that A/B has,
but does NOT have any for [21,40] because it was down when those were written

Now a client slices [10,1000] with count=11. The co-ordinating node will 
reconcile that; C's tombstones will override A/B (I'm assuming tombstones are 
later than A+B's columns), but since C is lacking the remainder of columns 
you don't just get some columns at lowered consistency level - you actually get 
a short result, and the application or high-level client will believe that 
the iteration is complete.

This was the primary reason why I said in the OP that I believed iterating 
over columns is impossible to do reliably with QUORUM. I somehow lost this 
when re-phrasing the JIRA post a couple of times.

Note: The short read case is not something I have tested and triggered, so is 
based on extrapolation from my understanding of the code.



 read repair/reconciliation breaks slice based iteration at QUORUM
 -

 Key: CASSANDRA-2643
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2643
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.5
Reporter: Peter Schuller
Priority: Critical
 Attachments: slicetest.py


 In short, I believe iterating over columns is impossible to do reliably with 
 QUORUM due to the way reconciliation works.
 The problem is that the SliceQueryFilter is executing locally when reading on 
 a node, but no attempts seem to be made to consider limits when doing 
 reconciliation and/or read-repair (RowRepairResolver.resolveSuperset() and 
 ColumnFamily.resolve()).
 If a node slices and comes up with 100 columns, and another node slices and 
 comes up with 100 columns, some of which are unique to each side, 
 reconciliation results in  100 columns in the result set. In this case the 
 effect is limited to client gets more than asked for, but the columns still 
 accurately represent the range. This is easily triggered by my test-case.
 In addition to the client receiving too many columns, I believe some of 
 them will not be satisfying the QUORUM consistency level for the same reasons 
 as with deletions (see discussion below).
 Now, there *should* be a problem for tombstones as well, but it's more 
 subtle. Suppose A has:
   1
   2
   3
   4
   5
   6
 and B has:
   1
   del 2
   del 3
   del 4
   5
   6 
 If you now slice 1-6 with count=3 the tombstones from B will reconcile with 
 those from A - fine. So you end up getting 1,5,6 back. This made it a bit 
 difficult to trigger in a test case until I realized what was going on. At 
 first I was hoping to see a short iteration result, which would mean that 
 the process of iterating until you get a short result will cause spurious 
 end of columns and thus make it impossible to iterate correctly.
 So; due to 5-6 existing (and if they didn't, you legitimately reached 
 end-of-columns) we do indeed get a result of size 3 which contains 1,5 and 6. 
 However, only node B would have contributed columns 5 and 6; so there is 
 actually no QUORUM consistency on the co-ordinating node with respect to 
 these columns. If node A and C also had 5 and 6, they would not have been 
 considered.
 Am I wrong?
 In any case; using script I'm about to attach, you can trigger the 
 over-delivery case very easily:
 (0) disable hinted hand-off to avoid that interacting with the test
 (1) start three nodes
 (2) create ks 'test' with rf=3 and cf 'slicetest'
 (3) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (4) stop node A
 (5) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (6) start node A, wait for B and C to consider it up
 (7) ./slicetest.py hostname_of_node_A slice # make A co-ordinator though it 
 doesn't necessarily matter
 You can also pass 'delete' (random deletion of 50% of contents) or 
 'deleterange' (delete all in [0.2,0.8]) to slicetest, but you don't trigger a 
 short read by doing that (see discussion above).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2646) Cli should be able to specify a limit for get_slice

Cli should be able to specify a limit for get_slice
---

 Key: CASSANDRA-2646
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2646
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Affects Versions: 0.7.0
Reporter: Sylvain Lebresne
Priority: Minor


The cli doesn't allow
{noformat}
get cf['k'] limit 5;
{noformat}
but should. Actually it should probably allow
{noformat}
get cf['k']['c':'g'] limit 10;
{noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2646) Cli should be able to specify a limit for get_slice


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2646:


Attachment: 0001-Add-limit-for-get-when-a-slice.patch

Patch (against 0.7) that adds the limit part. If someone wants to build on this 
to add support for start/end, be my guest, but I need that patch for a script 
I'm about to attach CASSANDRA-2643 so that's before all a way to make this 
public.

 Cli should be able to specify a limit for get_slice
 ---

 Key: CASSANDRA-2646
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2646
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Affects Versions: 0.7.0
Reporter: Sylvain Lebresne
Priority: Minor
  Labels: cli
 Attachments: 0001-Add-limit-for-get-when-a-slice.patch


 The cli doesn't allow
 {noformat}
 get cf['k'] limit 5;
 {noformat}
 but should. Actually it should probably allow
 {noformat}
 get cf['k']['c':'g'] limit 10;
 {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1102773 - /cassandra/branches/cassandra-0.8/doc/cql/CQL.html

2011-05-13 Thread eevans

Author: eevans
Date: Fri May 13 14:56:50 2011
New Revision: 1102773

URL: http://svn.apache.org/viewvc?rev=1102773view=rev
Log:
regenerate html documentation

Modified:
cassandra/branches/cassandra-0.8/doc/cql/CQL.html

Modified: cassandra/branches/cassandra-0.8/doc/cql/CQL.html
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/doc/cql/CQL.html?rev=1102773r1=1102772r2=1102773view=diff
==
--- cassandra/branches/cassandra-0.8/doc/cql/CQL.html (original)
+++ cassandra/branches/cassandra-0.8/doc/cql/CQL.html Fri May 13 14:56:50 2011
@@ -1,25 +1,4 @@
-!--
- 
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements.  See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership.  The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- License); you may not use this file except in compliance
- with the License.  You may obtain a copy of the License at
- 
-   http://www.apache.org/licenses/LICENSE-2.0
- 
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied.  See the License for the
- specific language governing permissions and limitations
- under the License.
- 
---
-
-?xml version='1.0' encoding='utf-8' ?!DOCTYPE html PUBLIC -//W3C//DTD 
XHTML 1.0 Transitional//EN 
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;html 
xmlns=http://www.w3.org/1999/xhtml;headmeta http-equiv=Content-Type 
content=text/html; charset=utf-8//headbodyh1 
id=CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) 
v1.0.0/h1h2 id=TableofContentsTable of Contents/h2ol 
style=list-style: none;lia 
href=#CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) 
v1.0.0/aol style=list-style: none;lia href=#TableofContentsTable of 
Contents/a/lilia href=#USEUSE/a/lilia 
href=#SELECTSELECT/aol style=list-style: none;lia 
href=#SpecifyingColumnsSpecifying Columns/a/lilia 
href=#ColumnFamilyColumn Family/a/lilia 
href=#ConsistencyLevelConsistency Level/a/lilia 
href=#FilteringrowsFiltering rows/a/lilia 
href=#LimitsLimits/a/li/ol/lili
 a href=#UPDATEUPDATE/aol style=list-style: none;lia 
href=#ColumnFamily2Column Family/a/lilia 
href=#ConsistencyLevel2Consistency Level/a/lilia 
href=#SpecifyingColumnsandRowSpecifying Columns and 
Row/a/li/ol/lilia href=#DELETEDELETE/aol style=list-style: 
none;lia href=#SpecifyingColumns2Specifying Columns/a/lilia 
href=#ColumnFamily3Column Family/a/lilia 
href=#ConsistencyLevel3Consistency Level/a/lilia 
href=#deleterowsSpecifying Rows/a/li/ol/lilia 
href=#TRUNCATETRUNCATE/a/lilia href=#CREATEKEYSPACECREATE 
KEYSPACE/a/lilia href=#CREATECOLUMNFAMILYCREATE COLUMNFAMILY/aol 
style=list-style: none;lia href=#keytypesSpecifying Key 
Type/a/lilia href=#columntypesSpecifying Column Type 
(optional)/a/lilia href=#ColumnFamilyOptionsoptionalColumn Family 
Options (optional)/a/li/ol/lilia href=#CREATEINDEXCREATE 
INDEX/a/li
 lia href=#DROPDROP/a/lilia href=#CommonIdiomsCommon 
 Idioms/aol style=list-style: none;lia href=#consistencySpecifying 
 Consistency/a/lilia href=#termsTerm 
 specification/a/li/ol/li/ol/lilia 
 href=#VersioningVersioning/a/lilia 
 href=#ChangesChanges/a/li/olh2 
 id=USEUSE/h2piSynopsis:/i/pprecodeUSE lt;KEYSPACEgt;;
+?xml version='1.0' encoding='utf-8' ?!DOCTYPE html PUBLIC -//W3C//DTD 
XHTML 1.0 Transitional//EN 
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;html 
xmlns=http://www.w3.org/1999/xhtml;headmeta http-equiv=Content-Type 
content=text/html; charset=utf-8//headbodyh1 
id=CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) 
v1.0.0/h1h2 id=TableofContentsTable of Contents/h2ol 
style=list-style: none;lia 
href=#CassandraQueryLanguageCQLv1.0.0Cassandra Query Language (CQL) 
v1.0.0/aol style=list-style: none;lia href=#TableofContentsTable of 
Contents/a/lilia href=#USEUSE/a/lilia 
href=#SELECTSELECT/aol style=list-style: none;lia 
href=#SpecifyingColumnsSpecifying Columns/a/lilia 
href=#ColumnFamilyColumn Family/a/lilia 
href=#ConsistencyLevelConsistency Level/a/lilia 
href=#FilteringrowsFiltering rows/a/lilia 
href=#LimitsLimits/a/li/ol/lili
 a href=#INSERTINSERT/a/lilia href=#UPDATEUPDATE/aol 
style=list-style: none;lia href=#ColumnFamily2Column 
Family/a/lilia href=#ConsistencyLevel2Consistency 
Level/a/lilia href=#SpecifyingColumnsandRowSpecifying Columns and 
Row/a/li/ol/lilia href=#DELETEDELETE/aol style=list-style: 
none;lia href=#SpecifyingColumns2Specifying Columns/a/lilia 
href=#ColumnFamily3Column Family/a/lilia 
href=#ConsistencyLevel3Consistency Level/a/lilia 
href=#deleterowsSpecifying Rows/a/li/ol/lilia 
href=#TRUNCATETRUNCATE/a/lilia href=#CREATEKEYSPACECREATE 
KEYSPACE/a/lilia href=#CREATECOLUMNFAMILYCREATE COLUMNFAMILY/aol 
style=list-style: none;lia

[jira] [Updated] (CASSANDRA-2643) read repair/reconciliation breaks slice based iteration at QUORUM


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2643:


Attachment: short_read.sh

You are right, there is a problem here.

I'll just note that you example is not a good example for QUORUM, because the 
fact that only C has [10,20] of column deletions means this situation did not 
happen with QUORUM writes (and the consistency guarantee for QUORUM involves 
read and write being at QUORUM).

However, this still show that there is a problem for ONE (writes) + ALL 
(reads). And it's not hard to come up with an example for QUORUM (reads and 
writes). Just consider the case where you insert like 10 columns and then 
delete the 3 first ones but with each time 1 node down, but never the same one.

To make this concrete, I'm attaching a script that produce this short read 
effect. Disclaimer: it uses https://github.com/pcmanus/ccm and require the 
patch I've attached to CASSANDRA-2646 (to be able to do a bounded slice with 
the cli).

The simplest way to fix that I see (which doesn't imply simple per se) would be 
to requests more columns if we're short after a resolve on the coordinator.  
Yes in theory it means we may have to do a unknown number of such re-request, 
but in practice I strongly doubt this will be a problem. The problem has very 
little chance to happen in real life to start with (for QUORUM, my script is 
simple but implements something that has very very little change to actually 
happen in real life -- especially with HH, read repair and repair), but the 
chances that if that happens we need more that 1 re-request are ridiculously 
small.


 read repair/reconciliation breaks slice based iteration at QUORUM
 -

 Key: CASSANDRA-2643
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2643
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.5
Reporter: Peter Schuller
Priority: Critical
 Attachments: short_read.sh, slicetest.py


 In short, I believe iterating over columns is impossible to do reliably with 
 QUORUM due to the way reconciliation works.
 The problem is that the SliceQueryFilter is executing locally when reading on 
 a node, but no attempts seem to be made to consider limits when doing 
 reconciliation and/or read-repair (RowRepairResolver.resolveSuperset() and 
 ColumnFamily.resolve()).
 If a node slices and comes up with 100 columns, and another node slices and 
 comes up with 100 columns, some of which are unique to each side, 
 reconciliation results in  100 columns in the result set. In this case the 
 effect is limited to client gets more than asked for, but the columns still 
 accurately represent the range. This is easily triggered by my test-case.
 In addition to the client receiving too many columns, I believe some of 
 them will not be satisfying the QUORUM consistency level for the same reasons 
 as with deletions (see discussion below).
 Now, there *should* be a problem for tombstones as well, but it's more 
 subtle. Suppose A has:
   1
   2
   3
   4
   5
   6
 and B has:
   1
   del 2
   del 3
   del 4
   5
   6 
 If you now slice 1-6 with count=3 the tombstones from B will reconcile with 
 those from A - fine. So you end up getting 1,5,6 back. This made it a bit 
 difficult to trigger in a test case until I realized what was going on. At 
 first I was hoping to see a short iteration result, which would mean that 
 the process of iterating until you get a short result will cause spurious 
 end of columns and thus make it impossible to iterate correctly.
 So; due to 5-6 existing (and if they didn't, you legitimately reached 
 end-of-columns) we do indeed get a result of size 3 which contains 1,5 and 6. 
 However, only node B would have contributed columns 5 and 6; so there is 
 actually no QUORUM consistency on the co-ordinating node with respect to 
 these columns. If node A and C also had 5 and 6, they would not have been 
 considered.
 Am I wrong?
 In any case; using script I'm about to attach, you can trigger the 
 over-delivery case very easily:
 (0) disable hinted hand-off to avoid that interacting with the test
 (1) start three nodes
 (2) create ks 'test' with rf=3 and cf 'slicetest'
 (3) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (4) stop node A
 (5) ./slicetest.py hostname_of_node_C insert # let it run for a few seconds, 
 then ctrl-c
 (6) start node A, wait for B and C to consider it up
 (7) ./slicetest.py hostname_of_node_A slice # make A co-ordinator though it 
 doesn't necessarily matter
 You can also pass 'delete' (random deletion of 50% of contents) or 
 'deleterange' (delete all in [0.2,0.8]) to slicetest, but you don't trigger a 
 short read by doing that (see discussion above).

--
This

[jira] [Updated] (CASSANDRA-2646) Cli should be able to specify a limit for get_slice


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2646:
--

Reviewer: xedin
Assignee: Sylvain Lebresne

 Cli should be able to specify a limit for get_slice
 ---

 Key: CASSANDRA-2646
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2646
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Affects Versions: 0.7.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Priority: Minor
  Labels: cli
 Attachments: 0001-Add-limit-for-get-when-a-slice.patch


 The cli doesn't allow
 {noformat}
 get cf['k'] limit 5;
 {noformat}
 but should. Actually it should probably allow
 {noformat}
 get cf['k']['c':'g'] limit 10;
 {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2646) Cli should be able to specify a limit for get_slice

2011-05-13 Thread Pavel Yaskevich (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033077#comment-13033077
 ] 

Pavel Yaskevich commented on CASSANDRA-2646:


+1

 Cli should be able to specify a limit for get_slice
 ---

 Key: CASSANDRA-2646
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2646
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Affects Versions: 0.7.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Priority: Minor
  Labels: cli
 Attachments: 0001-Add-limit-for-get-when-a-slice.patch


 The cli doesn't allow
 {noformat}
 get cf['k'] limit 5;
 {noformat}
 but should. Actually it should probably allow
 {noformat}
 get cf['k']['c':'g'] limit 10;
 {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1102793 - /cassandra/branches/cassandra-0.8/drivers/py/cqlsh

2011-05-13 Thread jbellis

Author: jbellis
Date: Fri May 13 15:34:51 2011
New Revision: 1102793

URL: http://svn.apache.org/viewvc?rev=1102793view=rev
Log:
fix cqlsh

Modified:
cassandra/branches/cassandra-0.8/drivers/py/cqlsh

Modified: cassandra/branches/cassandra-0.8/drivers/py/cqlsh
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/drivers/py/cqlsh?rev=1102793r1=1102792r2=1102793view=diff
==
--- cassandra/branches/cassandra-0.8/drivers/py/cqlsh (original)
+++ cassandra/branches/cassandra-0.8/drivers/py/cqlsh Fri May 13 15:34:51 2011
@@ -30,7 +30,7 @@ try:
 except ImportError:
 sys.path.append(os.path.abspath(os.path.dirname(__file__)))
 import cql
-from cql.results import ResultSet
+from cql.cursor import _COUNT_DESCRIPTION
 
 HISTORY = os.path.join(os.path.expanduser('~'), '.cqlsh')
 CQLTYPES = (bytes, ascii, utf8, timeuuid, uuid, long, int)
@@ -131,7 +131,9 @@ class Shell(cmd.Cmd):
 
 self.cursor.execute(statement)
 
-if isinstance(self.cursor.result, ResultSet):
+if self.cursor.description is _COUNT_DESCRIPTION:
+if self.cursor.result: print self.cursor.result[0]
+else:
 for x in range(self.cursor.rowcount):
 row = self.cursor.fetchone()
 self.printout(repr(row[0]), BLUE, False)
@@ -142,8 +144,6 @@ class Shell(cmd.Cmd):
 self.printout(,, newline=False)
 self.printout(repr(value), YELLOW, False)
 self.printout()
-else:
-if self.cursor.result: print self.cursor.result[0]
 
 def emptyline(self):
 pass

[jira] [Commented] (CASSANDRA-2640) `ant clean build test` is broken

2011-05-13 Thread Stephen Connolly (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033094#comment-13033094
 ] 

Stephen Connolly commented on CASSANDRA-2640:
-

Datapoint: Works with Apache Ant(TM) version 1.8.2 compiled on February 28 2011

 `ant clean build test` is broken
 

 Key: CASSANDRA-2640
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2640
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.5, 0.8.0 beta 2
Reporter: Gary Dusbabek
Assignee: Stephen Connolly

 `ant clean build` works.
 `ant test` works.
 `ant clean build test` is broken.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2640) `ant clean build test` is broken

2011-05-13 Thread Stephen Connolly (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033097#comment-13033097
 ] 

Stephen Connolly commented on CASSANDRA-2640:
-

Also works with ant 1.8.1
The issue seems to be ant 1.7.1

 `ant clean build test` is broken
 

 Key: CASSANDRA-2640
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2640
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.5, 0.8.0 beta 2
Reporter: Gary Dusbabek
Assignee: Stephen Connolly

 `ant clean build` works.
 `ant test` works.
 `ant clean build test` is broken.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CASSANDRA-2640) `ant clean build test` is broken

2011-05-13 Thread Gary Dusbabek (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Dusbabek resolved CASSANDRA-2640.
--

Resolution: Not A Problem

Dear ppls,  Please upgrade your ants.  That is all.  Thanks you.

 `ant clean build test` is broken
 

 Key: CASSANDRA-2640
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2640
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.5, 0.8.0 beta 2
Reporter: Gary Dusbabek
Assignee: Stephen Connolly

 `ant clean build` works.
 `ant test` works.
 `ant clean build test` is broken.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2530) Additional AbstractType data type definitions to enrich CQL

2011-05-13 Thread Rick Shaw (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rick Shaw updated CASSANDRA-2530:
-

Attachment: patch-to-add-4-new-AbstractTypes-and-CQL-support-v4.txt

- rebased to RC1
- Dropped {{BytesType}}
- Added CQL modifications for new data types

 Additional AbstractType data type definitions to enrich CQL
 ---

 Key: CASSANDRA-2530
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2530
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.8.0 beta 2
Reporter: Rick Shaw
Priority: Trivial
  Labels: cql
 Attachments: patch-to-add -5 -new-AbstractTypes-v2.txt, patch-to-add 
 -5 -new-AbstractTypes-v3.txt, patch-to-add -5 -new-AbstractTypes.txt, 
 patch-to-add-4-new-AbstractTypes-and-CQL-support-v4.txt


 Provide 5 additional Datatypes: ByteType, DateType, BooleanType, FloatType, 
 DoubleType.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2530) Additional AbstractType data type definitions to enrich CQL

2011-05-13 Thread Rick Shaw (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033119#comment-13033119
 ] 

Rick Shaw commented on CASSANDRA-2530:
--

Oops... meant {{ByteType}}.

 Additional AbstractType data type definitions to enrich CQL
 ---

 Key: CASSANDRA-2530
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2530
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.8.0 beta 2
Reporter: Rick Shaw
Priority: Trivial
  Labels: cql
 Attachments: patch-to-add -5 -new-AbstractTypes-v2.txt, patch-to-add 
 -5 -new-AbstractTypes-v3.txt, patch-to-add -5 -new-AbstractTypes.txt, 
 patch-to-add-4-new-AbstractTypes-and-CQL-support-v4.txt


 Provide 5 additional Datatypes: ByteType, DateType, BooleanType, FloatType, 
 DoubleType.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2647) Cassandra can't find jamm on startup

2011-05-13 Thread Marcus Bointon (JIRA)

Cassandra can't find jamm on startup


 Key: CASSANDRA-2647
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2647
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
 Environment: Ubuntu 10.04 Lucid
Reporter: Marcus Bointon


I installed the Debian package (from 
http://www.apache.org/dist/cassandra/debian unstable) of Cassandra 0.8beta2 on 
Ubuntu 10.04 with the sun jdk over a working copy of 0.7.2. It broke on restart.
On startup it gives this:
{code}
Error occurred during initialization of VM
agent library failed to init: instrument
Error opening zip file or JAR manifest missing : /lib/jamm-0.2.2.jar
{code}
/etc/cassandra/cassandra-env.sh contains this:
{code}
# add the jamm javaagent
check_openjdk=$(java -version 21 | awk '{if (NR == 2) {print $1}}')
if [ $check_openjdk != OpenJDK ]
then
JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.2.jar
fi
{code}
By default, CASSANDRA_HOME is not set, so it's looking in /lib for this jar. It 
seems CASSANDRA_HOME should be set to /usr/share/cassandra, since that's where 
jamm-0.2.2.jar is installed, but that means the path is still wrong.

I set CASSANDRA_HOME to /usr/share/cassandra and changed the JVM_OPTS line to 
this:
{code}
JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/jamm-0.2.2.jar
{code}

and then cassandra started ok.

Is this a bug or did I miss something?

I also noticed that this line appears to be the only use of CASSANDRA_HOME.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1610) Pluggable Compaction

[
https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033124#comment-13033124
]

Sylvain Lebresne commented on CASSANDRA-1610:
-

Looking quickly through that code, it looks a good chunk of the code is here to
support the expiring of sstables, and it's pretty much hardcoded. Isn't there a
way to encapsulate that better ? All the part about collecting and keeping the
max timestamp in a sstable also doesn't really strike me as making the
compaction more pluggable.

I'm not necessarily saying the TimestampBucketedCompactionStrategy is not
useful (but I'm not saying it is either), but at the very least I think that
what this patch does should be redefined clearly (it's not what I call making
compaction pluggable, at least not just that), and I'm pretty sure it does
multiple think and that I'm not necessary ok with all these things equally.

Pluggable Compaction

Key: CASSANDRA-1610
URL: https://issues.apache.org/jira/browse/CASSANDRA-1610
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Chris Goffinet
Assignee: Alan Liang
Priority: Minor
Labels: compaction
Fix For: 1.0

Attachments: 0001-move-compaction-code-into-own-package.patch,
0002-Pluggable-Compaction-and-Expiration.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled

2011-05-13 Thread T Jake Luciani (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033147#comment-13033147
 ] 

T Jake Luciani commented on CASSANDRA-2034:
---

Don't hints timeout?  would there be a chance of never resolving the 
discrepancy with this approach?

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
 Fix For: 1.0

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1610) Pluggable Compaction

[
https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033152#comment-13033152
]

Stu Hood commented on CASSANDRA-1610:
-

bq. but at the very least I think that what this patch does should be redefined
clearly (it's not what I call making compaction pluggable, at least not just
that), and I'm pretty sure it does multiple think and that I'm not necessary ok
with all these things equally.
Agreed. Alan: would you mind overhauling the description of this ticket to
better describe the goals?

As usual, the problem with making something pluggable is that you need at least
2 implementations of the interface to make it worthwhile. We could separate out
the TimestampBucketedCompactionStrategy and its required additions into a
separate ticket, but it would be a mostly symbolic split. One split that might
be reasonable (but which I'm sure Alan would prefer to avoid) would be to add
TimeBucketed in this ticket, but to split out implementing expiration into a
separate ticket.

Pluggable Compaction

Attachments: 0001-move-compaction-code-into-own-package.patch,
0002-Pluggable-Compaction-and-Expiration.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2608) CliTest fails occasionally

2011-05-13 Thread Pavel Yaskevich (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033158#comment-13033158
 ] 

Pavel Yaskevich commented on CASSANDRA-2608:


I tried to reproduce this behavior few days in a row by running tests each hour 
on linux/mac and couldn't get it failing this way, it could be related to 
EmbeddedCassandraService failing mysteriously on your box...

 CliTest fails occasionally
 --

 Key: CASSANDRA-2608
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2608
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
Reporter: Jonathan Ellis
Assignee: Pavel Yaskevich
Priority: Minor
 Fix For: 0.8.1


 I saw this just now, but re-run of the test passes:
 {noformat}
 [junit] Testcase: testCli(org.apache.cassandra.cli.CliTest):  FAILED
 [junit] null
 [junit]  processing drop keyspace tesTIN; expected:[] but was:[null
 [junit] ]
 [junit] junit.framework.AssertionFailedError: null
 [junit]  processing drop keyspace tesTIN; expected:[] but was:[null
 [junit] ]
 [junit]   at org.apache.cassandra.cli.CliTest.testCli(CliTest.java:203)
 {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled

[
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033159#comment-13033159
]

Stu Hood commented on CASSANDRA-2034:
-

bq. Better would be to add a hook to messagingservice callback expiration, and
fire hint recording from there
bq. So I think what we want to do, with this option on, is to attempt the hint
write but if we can't do it in a reasonable time...
+1. An expiration handler on the messaging service queue, and throttled local
hint writing should work very well.

Make Read Repair unnecessary when Hinted Handoff is enabled
---

Key: CASSANDRA-2034
URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Fix For: 1.0

Original Estimate: 8h
Remaining Estimate: 8h

Currently, HH is purely an optimization -- if a machine goes down, enabling
HH means RR/AES will have less work to do, but you can't disable RR entirely
in most situations since HH doesn't kick in until the FailureDetector does.
Let's add a scheduled task to the mutate path, such that we return to the
client normally after ConsistencyLevel is achieved, but after RpcTimeout we
check the responseHandler write acks and write local hints for any missing
targets.
This would making disabling RR when HH is enabled a much more reasonable
option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1102850 - /cassandra/branches/cassandra-0.8/drivers/py/setup.py

2011-05-13 Thread eevans

Author: eevans
Date: Fri May 13 18:19:31 2011
New Revision: 1102850

URL: http://svn.apache.org/viewvc?rev=1102850view=rev
Log:
updated versioning for Python driver

Modified:
cassandra/branches/cassandra-0.8/drivers/py/setup.py

Modified: cassandra/branches/cassandra-0.8/drivers/py/setup.py
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/drivers/py/setup.py?rev=1102850r1=1102849r2=1102850view=diff
==
--- cassandra/branches/cassandra-0.8/drivers/py/setup.py (original)
+++ cassandra/branches/cassandra-0.8/drivers/py/setup.py Fri May 13 18:19:31 
2011
@@ -20,7 +20,7 @@ from os.path import abspath, join, dirna
 
 setup(
 name=cql,
-version=1.0.1,
+version=1.0.2,
 description=Cassandra Query Language driver,
 long_description=open(abspath(join(dirname(__file__), 'README'))).read(),
 url=http://cassandra.apache.org;,

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled

[
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033202#comment-13033202
]

Jonathan Ellis commented on CASSANDRA-2034:
---

bq. Don't hints timeout?

No (but cleanup can purge hinted rows, so don't do that unless all hints have
been replayed).

bq. would there be a chance of never resolving the discrepancy with this
approach?

Definitely in the case of a node went down, so I wrote some hints, but then
the hinted node lost its hdd too. In that case you'd need to run AE repair.

So the idea is not to make AE repair (completely) obsolete, only RR.

Make Read Repair unnecessary when Hinted Handoff is enabled
---

Original Estimate: 8h
Remaining Estimate: 8h

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CASSANDRA-2608) CliTest fails occasionally


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis resolved CASSANDRA-2608.
---

   Resolution: Cannot Reproduce
Fix Version/s: (was: 0.8.1)
 Assignee: (was: Pavel Yaskevich)

 CliTest fails occasionally
 --

 Key: CASSANDRA-2608
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2608
 Project: Cassandra
  Issue Type: Bug
  Components: Tests
Reporter: Jonathan Ellis
Priority: Minor

 I saw this just now, but re-run of the test passes:
 {noformat}
 [junit] Testcase: testCli(org.apache.cassandra.cli.CliTest):  FAILED
 [junit] null
 [junit]  processing drop keyspace tesTIN; expected:[] but was:[null
 [junit] ]
 [junit] junit.framework.AssertionFailedError: null
 [junit]  processing drop keyspace tesTIN; expected:[] but was:[null
 [junit] ]
 [junit]   at org.apache.cassandra.cli.CliTest.testCli(CliTest.java:203)
 {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2647) Cassandra can't find jamm on startup


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2647:
--

 Reviewer: brandon.williams
 Priority: Minor  (was: Major)
Affects Version/s: 0.8 beta 1
Fix Version/s: 0.8.1
 Assignee: paul cannon

 Cassandra can't find jamm on startup
 

 Key: CASSANDRA-2647
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2647
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
Affects Versions: 0.8 beta 1
 Environment: Ubuntu 10.04 Lucid
Reporter: Marcus Bointon
Assignee: paul cannon
Priority: Minor
 Fix For: 0.8.1


 I installed the Debian package (from 
 http://www.apache.org/dist/cassandra/debian unstable) of Cassandra 0.8beta2 
 on Ubuntu 10.04 with the sun jdk over a working copy of 0.7.2. It broke on 
 restart.
 On startup it gives this:
 {code}
 Error occurred during initialization of VM
 agent library failed to init: instrument
 Error opening zip file or JAR manifest missing : /lib/jamm-0.2.2.jar
 {code}
 /etc/cassandra/cassandra-env.sh contains this:
 {code}
 # add the jamm javaagent
 check_openjdk=$(java -version 21 | awk '{if (NR == 2) {print $1}}')
 if [ $check_openjdk != OpenJDK ]
 then
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.2.jar
 fi
 {code}
 By default, CASSANDRA_HOME is not set, so it's looking in /lib for this jar. 
 It seems CASSANDRA_HOME should be set to /usr/share/cassandra, since that's 
 where jamm-0.2.2.jar is installed, but that means the path is still wrong.
 I set CASSANDRA_HOME to /usr/share/cassandra and changed the JVM_OPTS line to 
 this:
 {code}
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/jamm-0.2.2.jar
 {code}
 and then cassandra started ok.
 Is this a bug or did I miss something?
 I also noticed that this line appears to be the only use of CASSANDRA_HOME.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2411) log why a SSTable is being deleted

2011-05-13 Thread Robert Coli (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033221#comment-13033221
 ] 

Robert Coli commented on CASSANDRA-2411:


Looks good to me. +1. :D

 log why a SSTable is being deleted
 --

 Key: CASSANDRA-2411
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2411
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Robert Coli
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.8.1

 Attachments: 2411.txt


 ./src/java/org/apache/cassandra/io/sstable/SSTable.java: 147
 Has logger.info(Deleted  + desc); .
 This combined with the JVM not being able to delete files until a GC has run 
 means that restarting a node usually prompts a flood of log messages like :
 Deleted /mnt/var/cassandra/data/Digg/UserActivity-10658-Data.db
 I believe that I am not the only operator who reads a log upon restart that 
 says I deleted your data files and becomes concerned.
 Now, personally, I have read the code and understand the conditions under 
 which these files are deleted and so no longer get frightened. For new 
 operators and people who may feel less comfortable reading the code, 
 specifying WHY the file has been deleted (Deleted filename because it was 
 obsolete and marked for deletion as a result of compaction.) seems likely to 
 reduce blood pressure and operator stress levels. :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1102872 - /cassandra/branches/cassandra-0.8/conf/cassandra.yaml

2011-05-13 Thread jbellis

Author: jbellis
Date: Fri May 13 19:12:20 2011
New Revision: 1102872

URL: http://svn.apache.org/viewvc?rev=1102872view=rev
Log:
add example of what comma-delimited means to seeds line
patch by Joaquin Casares

Modified:
cassandra/branches/cassandra-0.8/conf/cassandra.yaml

Modified: cassandra/branches/cassandra-0.8/conf/cassandra.yaml
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/conf/cassandra.yaml?rev=1102872r1=1102871r2=1102872view=diff
==
--- cassandra/branches/cassandra-0.8/conf/cassandra.yaml (original)
+++ cassandra/branches/cassandra-0.8/conf/cassandra.yaml Fri May 13 19:12:20 
2011
@@ -102,6 +102,7 @@ seed_provider:
 - class_name: org.apache.cassandra.locator.SimpleSeedProvider
   parameters:
   # seeds is actually a comma-delimited list of addresses.
+  # Ex: ip1,ip2,ip3
   - seeds: 127.0.0.1
 
 # emergency pressure valve: each time heap usage after a full (CMS)

svn commit: r1102873 - in /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra: db/ColumnFamilyStore.java io/sstable/SSTable.java

2011-05-13 Thread jbellis

Author: jbellis
Date: Fri May 13 19:16:09 2011
New Revision: 1102873

URL: http://svn.apache.org/viewvc?rev=1102873view=rev
Log:
switch purge-compacted-sstables-at-startup logging to DEBUG level
patch by jbellis; reviewed by rcoli for CASSANDRA-2411

Modified:

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTable.java

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java?rev=1102873r1=1102872r2=1102873view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
 Fri May 13 19:16:09 2011
@@ -461,6 +461,7 @@ public class ColumnFamilyStore implement
  */
 public static void scrubDataDirectories(String table, String columnFamily)
 {
+logger.info(Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable));
 for (Map.EntryDescriptor,SetComponent sstableFiles : files(table, 
columnFamily, true).entrySet())
 {
 Descriptor desc = sstableFiles.getKey();

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTable.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTable.java?rev=1102873r1=1102872r2=1102873view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTable.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTable.java
 Fri May 13 19:16:09 2011
@@ -156,7 +156,7 @@ public abstract class SSTable
 {
 throw new IOError(e);
 }
-logger.info(Deleted  + desc);
+logger.debug(Deleted {}, desc);
 return true;
 }

[jira] [Commented] (CASSANDRA-2611) static block in AbstractCassandraDaemon makes it difficult to change log4j behavoiur


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033230#comment-13033230
 ] 

Jonathan Ellis commented on CASSANDRA-2611:
---

Sounds reasonable

 static block in AbstractCassandraDaemon makes it difficult to change log4j 
 behavoiur
 

 Key: CASSANDRA-2611
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2611
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Affects Versions: 0.7.4, 0.7.5
 Environment: Windows 7
Reporter: Paul Loy
Priority: Minor
  Labels: daemon, initialisation, log4j

 We embed Cassandra in our application - mainly because our webservices are 
 such a thin layer on top of Cassandra that it really does not make sense for 
 us to have Cassandra in an external JVM. In 0.7.0 this was all fine. Now 
 upgrading to 0.7.5, there is a static block in AbstractCassandraDaemon. This 
 gets called when the class is loaded causing us issues as we have not 
 generated the log4j.properties file at this point in time.
 Can this not be a protected method that is called when 
 AbstractCassandraDaemon is constructed? That way a) I can control the 
 behaviour and b) my log4j.properties file will have been generated by then.
 Thanks.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2647) Cassandra can't find jamm on startup

2011-05-13 Thread Brandon Williams (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033246#comment-13033246
 ] 

Brandon Williams commented on CASSANDRA-2647:
-

CASSANDRA_HOME should be getting set in cassandra.in.sh which is the first 
thing sourced by bin/cassandra and /etc/init.d/cassandra

 Cassandra can't find jamm on startup
 

 Key: CASSANDRA-2647
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2647
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
Affects Versions: 0.8 beta 1
 Environment: Ubuntu 10.04 Lucid
Reporter: Marcus Bointon
Assignee: paul cannon
Priority: Minor
 Fix For: 0.8.1


 I installed the Debian package (from 
 http://www.apache.org/dist/cassandra/debian unstable) of Cassandra 0.8beta2 
 on Ubuntu 10.04 with the sun jdk over a working copy of 0.7.2. It broke on 
 restart.
 On startup it gives this:
 {code}
 Error occurred during initialization of VM
 agent library failed to init: instrument
 Error opening zip file or JAR manifest missing : /lib/jamm-0.2.2.jar
 {code}
 /etc/cassandra/cassandra-env.sh contains this:
 {code}
 # add the jamm javaagent
 check_openjdk=$(java -version 21 | awk '{if (NR == 2) {print $1}}')
 if [ $check_openjdk != OpenJDK ]
 then
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.2.jar
 fi
 {code}
 By default, CASSANDRA_HOME is not set, so it's looking in /lib for this jar. 
 It seems CASSANDRA_HOME should be set to /usr/share/cassandra, since that's 
 where jamm-0.2.2.jar is installed, but that means the path is still wrong.
 I set CASSANDRA_HOME to /usr/share/cassandra and changed the JVM_OPTS line to 
 this:
 {code}
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/jamm-0.2.2.jar
 {code}
 and then cassandra started ok.
 Is this a bug or did I miss something?
 I also noticed that this line appears to be the only use of CASSANDRA_HOME.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2648) Failing while starting Cassandar server : Corrupt (negative) value length encountered

2011-05-13 Thread Olivier Smadja (JIRA)

Failing while starting Cassandar server : Corrupt (negative) value length 
encountered
-

 Key: CASSANDRA-2648
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2648
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.4
 Environment: Linux
Reporter: Olivier Smadja


 INFO 16:50:35,307 reading saved cache 
/home/tapix/data/cassandra/data/saved_caches/tapix_prod-StreamPostsStatistics-KeyCache
 INFO 16:50:35,316 Opening 
/home/tapix/data/cassandra/data/tapix_prod/StreamPostsStatistics-f-121
 INFO 16:50:35,320 Opening 
/home/tapix/data/cassandra/data/tapix_prod/StreamPostsStatistics-f-120
 INFO 16:50:35,329 Opening 
/home/tapix/data/cassandra/data/tapix_prod/StreamLine-f-9
 INFO 16:50:35,352 Creating new commitlog segment 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1305316235352.log
 INFO 16:50:35,362 Replaying 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1303829569725.log, 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1303948043185.log, 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1304361402015.log, 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1304728796807.log, 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1305208776962.log, 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1305316204091.log
 INFO 16:50:35,536 Finished reading 
/home/tapix/data/cassandra/data/commitlog/CommitLog-1303829569725.log
ERROR 16:50:35,537 Exception encountered during startup.
java.io.IOException: Corrupt (negative) value length encountered
at 
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:269)
at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:380)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:253)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156)
at 
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173)
at 
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
at 
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
Exception encountered during startup.
java.io.IOException: Corrupt (negative) value length encountered
at 
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:269)
at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at 
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:380)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:253)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156)
at 
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173)
at 
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
at 
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2649) work-around schema disagreements from cqlsh

2011-05-13 Thread Eric Evans (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Evans updated CASSANDRA-2649:
--

Attachment: 
v1-0001-CASSANDRA-2649-try-to-work-around-schema-disagreement-.txt

 work-around schema disagreements from cqlsh
 ---

 Key: CASSANDRA-2649
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2649
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.8.0 beta 2
Reporter: Eric Evans
Assignee: Eric Evans
Priority: Minor
  Labels: cql
 Fix For: 0.8.0

 Attachments: 
 v1-0001-CASSANDRA-2649-try-to-work-around-schema-disagreement-.txt


 It is handy to be able to put CQL statements in a flat-file and load them by 
 redirecting to {{cqlsh}} stdin, but this can fail on a cluster when executing 
 statements that modify schema.
 The attached patch works around this problem by retrying up to 3 times, with 
 a progressive delay after each attempt.  A better solution would probably be 
 to compare schema versions, but this seems to work well enough, and is better 
 than _not_ handling it at all.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1610) Pluggable Compaction

2011-05-13 Thread Alan Liang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033258#comment-13033258
]

Alan Liang commented on CASSANDRA-1610:
---

Looking quickly through that code, it looks a good chunk of the code is here
to support the expiring of sstables, and it's pretty much hardcoded. Isn't
there a way to encapsulate that better ?

You're right, it might make more sense to allow a strategy to define how it
should expire the sstables.

I'll try and fix the description. But I want to keep the implemented strategies
with this ticket because they justify why the interfaces are worthwhile as Stu
pointed out above.

Pluggable Compaction

Attachments: 0001-move-compaction-code-into-own-package.patch,
0002-Pluggable-Compaction-and-Expiration.patch

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2649) work-around schema disagreements from cqlsh

2011-05-13 Thread Eric Evans (JIRA)

work-around schema disagreements from cqlsh
---

 Key: CASSANDRA-2649
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2649
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 0.8.0 beta 2
Reporter: Eric Evans
Assignee: Eric Evans
Priority: Minor
 Fix For: 0.8.0
 Attachments: 
v1-0001-CASSANDRA-2649-try-to-work-around-schema-disagreement-.txt

It is handy to be able to put CQL statements in a flat-file and load them by 
redirecting to {{cqlsh}} stdin, but this can fail on a cluster when executing 
statements that modify schema.

The attached patch works around this problem by retrying up to 3 times, with a 
progressive delay after each attempt.  A better solution would probably be to 
compare schema versions, but this seems to work well enough, and is better than 
_not_ handling it at all.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2411) log why a SSTable is being deleted

2011-05-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033260#comment-13033260
 ] 

Hudson commented on CASSANDRA-2411:
---

Integrated in Cassandra-0.8 #107 (See 
[https://builds.apache.org/hudson/job/Cassandra-0.8/107/])
switch purge-compacted-sstables-at-startup logging to DEBUG level
patch by jbellis; reviewed by rcoli for CASSANDRA-2411


 log why a SSTable is being deleted
 --

 Key: CASSANDRA-2411
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2411
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Robert Coli
Assignee: Jonathan Ellis
Priority: Minor
 Fix For: 0.8.1

 Attachments: 2411.txt


 ./src/java/org/apache/cassandra/io/sstable/SSTable.java: 147
 Has logger.info(Deleted  + desc); .
 This combined with the JVM not being able to delete files until a GC has run 
 means that restarting a node usually prompts a flood of log messages like :
 Deleted /mnt/var/cassandra/data/Digg/UserActivity-10658-Data.db
 I believe that I am not the only operator who reads a log upon restart that 
 says I deleted your data files and becomes concerned.
 Now, personally, I have read the code and understand the conditions under 
 which these files are deleted and so no longer get frightened. For new 
 operators and people who may feel less comfortable reading the code, 
 specifying WHY the file has been deleted (Deleted filename because it was 
 obsolete and marked for deletion as a result of compaction.) seems likely to 
 reduce blood pressure and operator stress levels. :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2034) Make Read Repair unnecessary when Hinted Handoff is enabled


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033273#comment-13033273
 ] 

Stu Hood commented on CASSANDRA-2034:
-

IMO, disabling RR entirely is never a good idea unless we are going to 
_guarantee_ hint delivery. But I agree that this ticket is a good idea because 
increasing the probability of hint delivery is healthy.

 Make Read Repair unnecessary when Hinted Handoff is enabled
 ---

 Key: CASSANDRA-2034
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2034
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
 Fix For: 1.0

   Original Estimate: 8h
  Remaining Estimate: 8h

 Currently, HH is purely an optimization -- if a machine goes down, enabling 
 HH means RR/AES will have less work to do, but you can't disable RR entirely 
 in most situations since HH doesn't kick in until the FailureDetector does.
 Let's add a scheduled task to the mutate path, such that we return to the 
 client normally after ConsistencyLevel is achieved, but after RpcTimeout we 
 check the responseHandler write acks and write local hints for any missing 
 targets.
 This would making disabling RR when HH is enabled a much more reasonable 
 option, which has a huge impact on read throughput.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2526) Add ant support to generate CQL documentation from textile source

2011-05-13 Thread Nate McCall (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nate McCall updated CASSANDRA-2526:
---

Attachment: 2526v2.txt

Accepted on maven central. 2526v2.txt Patch contains the correct maven 
coordinates for such.

 Add ant support to generate CQL documentation from textile source
 -

 Key: CASSANDRA-2526
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2526
 Project: Cassandra
  Issue Type: Task
Affects Versions: 0.8 beta 1
Reporter: Jonathan Ellis
Assignee: Nate McCall
Priority: Minor
 Fix For: 0.8.1

 Attachments: 2526.txt, 2526v2.txt


 possibly useful links: 
 - 
 http://help.eclipse.org/helios/topic/org.eclipse.mylyn.wikitext.help.ui/help/Markup-Conversion.html#ConversionusingAntbuildscripts
 - 
 http://www.mvnbrowser.com/artifact-details.html?groupId=org.eclipse.mylyn.wikitextartifactId=wikitext

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2647) Cassandra can't find jamm on startup

2011-05-13 Thread Marcus Bointon (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033287#comment-13033287
 ] 

Marcus Bointon commented on CASSANDRA-2647:
---

Thanks for that. I think I'd normally be looking for /etc/defaults/cassandra to 
set that kind of thing. Also, the supplied cassandra.in.sh does not set 
CASSANDRA_HOME, not even commented-out. It might be nice to mention that file 
in a comment in /etc/cassandra/cassandra.yaml

 Cassandra can't find jamm on startup
 

 Key: CASSANDRA-2647
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2647
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
Affects Versions: 0.8 beta 1
 Environment: Ubuntu 10.04 Lucid
Reporter: Marcus Bointon
Assignee: paul cannon
Priority: Minor
 Fix For: 0.8.1


 I installed the Debian package (from 
 http://www.apache.org/dist/cassandra/debian unstable) of Cassandra 0.8beta2 
 on Ubuntu 10.04 with the sun jdk over a working copy of 0.7.2. It broke on 
 restart.
 On startup it gives this:
 {code}
 Error occurred during initialization of VM
 agent library failed to init: instrument
 Error opening zip file or JAR manifest missing : /lib/jamm-0.2.2.jar
 {code}
 /etc/cassandra/cassandra-env.sh contains this:
 {code}
 # add the jamm javaagent
 check_openjdk=$(java -version 21 | awk '{if (NR == 2) {print $1}}')
 if [ $check_openjdk != OpenJDK ]
 then
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.2.jar
 fi
 {code}
 By default, CASSANDRA_HOME is not set, so it's looking in /lib for this jar. 
 It seems CASSANDRA_HOME should be set to /usr/share/cassandra, since that's 
 where jamm-0.2.2.jar is installed, but that means the path is still wrong.
 I set CASSANDRA_HOME to /usr/share/cassandra and changed the JVM_OPTS line to 
 this:
 {code}
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/jamm-0.2.2.jar
 {code}
 and then cassandra started ok.
 Is this a bug or did I miss something?
 I also noticed that this line appears to be the only use of CASSANDRA_HOME.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1608) Redesigned Compaction

2011-05-13 Thread Ryan King (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033294#comment-13033294
]

Ryan King commented on CASSANDRA-1608:
--

I only read the LevelDB stuff briefly. I think there's a lot we can learn, but
there's at least 2 challenges:

1) client supplied timestamps mean that you can't know that newer files
supercede older ones
2) the CF data model means that data for a given key in multiple sstables may
need to be merged

Redesigned Compaction
-

Key: CASSANDRA-1608
URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Chris Goffinet

After seeing the I/O issues in CASSANDRA-1470, I've been doing some more
thinking on this subject that I wanted to lay out.
I propose we redo the concept of how compaction works in Cassandra. At the
moment, compaction is kicked off based on a write access pattern, not read
access pattern. In most cases, you want the opposite. You want to be able to
track how well each SSTable is performing in the system. If we were to keep
statistics in-memory of each SSTable, prioritize them based on most accessed,
and bloom filter hit/miss ratios, we could intelligently group sstables that
are being read most often and schedule them for compaction. We could also
schedule lower priority maintenance on SSTable's not often accessed.
I also propose we limit the size of each SSTable to a fix sized, that gives
us the ability to better utilize our bloom filters in a predictable manner.
At the moment after a certain size, the bloom filters become less reliable.
This would also allow us to group data most accessed. Currently the size of
an SSTable can grow to a point where large portions of the data might not
actually be accessed as often.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-1610) Pluggable Compaction

2011-05-13 Thread Alan Liang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alan Liang updated CASSANDRA-1610:
--

Description:
In CASSANDRA-1608, I proposed some changes on how compaction works. I think it
also makes sense to allow the ability to have pluggable compaction per CF.
There could be many types of workloads where this makes sense. One example we
had at Digg was to completely throw away certain SSTables after N days.

The goal of this ticket is to make compaction pluggable enough to support
compaction based on max timestamp ordering of the sstables while satisfying max
sstable size, min and max compaction thresholds. Another goal is to allow
expiration of sstables based on a timestamp.

was:In CASSANDRA-1608, I proposed some changes on how compaction works. I
think it also makes sense to allow the ability to have pluggable compaction per
CF. There could be many types of workloads where this makes sense. One example
we had at Digg was to completely throw away certain SSTables after N days.

Pluggable Compaction

Attachments: 0001-move-compaction-code-into-own-package.patch,
0002-Pluggable-Compaction-and-Expiration.patch

In CASSANDRA-1608, I proposed some changes on how compaction works. I think
it also makes sense to allow the ability to have pluggable compaction per CF.
There could be many types of workloads where this makes sense. One example we
had at Digg was to completely throw away certain SSTables after N days.
The goal of this ticket is to make compaction pluggable enough to support
compaction based on max timestamp ordering of the sstables while satisfying
max sstable size, min and max compaction thresholds. Another goal is to allow
expiration of sstables based on a timestamp.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2650) AntiEntropyServiceCounterTest writes CounterUpdateColumn to disk

AntiEntropyServiceCounterTest writes CounterUpdateColumn to disk


 Key: CASSANDRA-2650
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2650
 Project: Cassandra
  Issue Type: Test
  Components: Tests
Reporter: Stu Hood
Priority: Minor
 Fix For: 1.0


By using an unwrapped RowMutation to apply a counter update, AESCT was 
applying/flushing a CounterUpdateColumn to disk. I'm still not a fan of the 
magic that is going on with CUC, but for now fixing misuses is the most 
reasonable approach.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2650) AntiEntropyServiceCounterTest writes CounterUpdateColumn to disk


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stu Hood updated CASSANDRA-2650:


Attachment: 0001-Make-IMutation-more-uniform-and-apply-IMutation-in-Uti.txt

 AntiEntropyServiceCounterTest writes CounterUpdateColumn to disk
 

 Key: CASSANDRA-2650
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2650
 Project: Cassandra
  Issue Type: Test
  Components: Tests
Reporter: Stu Hood
Priority: Minor
  Labels: counters
 Fix For: 1.0

 Attachments: 
 0001-Make-IMutation-more-uniform-and-apply-IMutation-in-Uti.txt


 By using an unwrapped RowMutation to apply a counter update, AESCT was 
 applying/flushing a CounterUpdateColumn to disk. I'm still not a fan of the 
 magic that is going on with CUC, but for now fixing misuses is the most 
 reasonable approach.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (CASSANDRA-2401) getColumnFamily() return null, which is not checked in ColumnFamilyStore.java scan() method, causing Timeout Exception in query

2011-05-13 Thread Tyler Hobbs (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs reopened CASSANDRA-2401:



With these changes, using a count of 0 in the SlicePredicate produces the 
following AssertionError (and a TimedOutExc for the client):

{noformat}
ERROR 16:13:38,864 Fatal exception in thread Thread[ReadStage:16,5,main]
java.lang.AssertionError: No data found for 
SliceQueryFilter(start=java.nio.HeapByteBuffer[pos=10 lim=10 cap=30], 
finish=java.nio.HeapByteBuffer[pos=17 lim=17 cap=30], reversed=false, count=0] 
in DecoratedKey(81509516161424251288255223397843705139, 
6b657931):QueryPath(columnFamilyName='cf', superColumnName='null', 
columnName='null') (original filter 
SliceQueryFilter(start=java.nio.HeapByteBuffer[pos=10 lim=10 cap=30], 
finish=java.nio.HeapByteBuffer[pos=17 lim=17 cap=30], reversed=false, count=0]) 
from expression 'cf.626972746864617465 EQ 1'
at 
org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1517)
at 
org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}

 getColumnFamily() return null, which is not checked in ColumnFamilyStore.java 
 scan() method, causing Timeout Exception in query
 ---

 Key: CASSANDRA-2401
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2401
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.7.0
 Environment: Hector 0.7.0-28, Cassandra 0.7.4, Windows 7, Eclipse
Reporter: Tey Kar Shiang
Assignee: Jonathan Ellis
 Fix For: 0.7.6

 Attachments: 2401-v2.txt, 2401-v3.txt, 2401.txt


 ColumnFamilyStore.java, line near 1680, ColumnFamily data = 
 getColumnFamily(new QueryFilter(dk, path, firstFilter)), the data is 
 returned null, causing NULL exception in satisfies(data, clause, primary) 
 which is not captured. The callback got timeout and return a Timeout 
 exception to Hector.
 The data is empty, as I traced, I have the the columns Count as 0 in 
 removeDeletedCF(), which return the null there. (I am new and trying to 
 understand the logics around still). Instead of crash to NULL, could we 
 bypass the data?
 About my test:
 A stress-test program to add, modify and delete data to keyspace. I have 30 
 threads simulate concurrent users to perform the actions above, and do a 
 query to all rows periodically. I have Column Family with rows (as File) and 
 columns as index (e.g. userID, fileType).
 No issue on the first day of test, and stopped for 3 days. I restart the test 
 on 4th day, 1 of the users failed to query the files (timeout exception 
 received). Most of the users are still okay with the query.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2401) getColumnFamily() return null, which is not checked in ColumnFamilyStore.java scan() method, causing Timeout Exception in query

2011-05-13 Thread Tyler Hobbs (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1301#comment-1301
 ] 

Tyler Hobbs edited comment on CASSANDRA-2401 at 5/13/11 9:54 PM:
-

With these changes, using a count of 0 in the SlicePredicate produces the 
following AssertionError (and a TimedOutExc for the client):

{noformat}
ERROR 16:13:38,864 Fatal exception in thread Thread[ReadStage:16,5,main]
java.lang.AssertionError: No data found for 
SliceQueryFilter(start=java.nio.HeapByteBuffer[pos=10 lim=10 cap=30], 
finish=java.nio.HeapByteBuffer[pos=17 lim=17 cap=30], reversed=false, count=0] 
in DecoratedKey(81509516161424251288255223397843705139, 
6b657931):QueryPath(columnFamilyName='cf', superColumnName='null', 
columnName='null') (original filter 
SliceQueryFilter(start=java.nio.HeapByteBuffer[pos=10 lim=10 cap=30], 
finish=java.nio.HeapByteBuffer[pos=17 lim=17 cap=30], reversed=false, count=0]) 
from expression 'cf.626972746864617465 EQ 1'
at 
org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1517)
at 
org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}

This was during a get_indexed_slices().

  was (Author: thobbs):
With these changes, using a count of 0 in the SlicePredicate produces the 
following AssertionError (and a TimedOutExc for the client):

{noformat}
ERROR 16:13:38,864 Fatal exception in thread Thread[ReadStage:16,5,main]
java.lang.AssertionError: No data found for 
SliceQueryFilter(start=java.nio.HeapByteBuffer[pos=10 lim=10 cap=30], 
finish=java.nio.HeapByteBuffer[pos=17 lim=17 cap=30], reversed=false, count=0] 
in DecoratedKey(81509516161424251288255223397843705139, 
6b657931):QueryPath(columnFamilyName='cf', superColumnName='null', 
columnName='null') (original filter 
SliceQueryFilter(start=java.nio.HeapByteBuffer[pos=10 lim=10 cap=30], 
finish=java.nio.HeapByteBuffer[pos=17 lim=17 cap=30], reversed=false, count=0]) 
from expression 'cf.626972746864617465 EQ 1'
at 
org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1517)
at 
org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
  
 getColumnFamily() return null, which is not checked in ColumnFamilyStore.java 
 scan() method, causing Timeout Exception in query
 ---

 Key: CASSANDRA-2401
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2401
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.7.0
 Environment: Hector 0.7.0-28, Cassandra 0.7.4, Windows 7, Eclipse
Reporter: Tey Kar Shiang
Assignee: Jonathan Ellis
 Fix For: 0.7.6

 Attachments: 2401-v2.txt, 2401-v3.txt, 2401.txt


 ColumnFamilyStore.java, line near 1680, ColumnFamily data = 
 getColumnFamily(new QueryFilter(dk, path, firstFilter)), the data is 
 returned null, causing NULL exception in satisfies(data, clause, primary) 
 which is not captured. The callback got timeout and return a Timeout 
 exception to Hector.
 The data is empty, as I traced, I have the the columns Count as 0 in 
 removeDeletedCF(), which return the null there. (I am new and trying to 
 understand the logics around still). Instead of crash to NULL, could we 
 bypass the data?
 About my test:
 A stress-test program to add, modify and delete data to keyspace. I have 30 
 threads simulate concurrent users to perform the actions above, and do a 
 query to all rows periodically. I have Column Family with rows (as File) and 
 columns as index (e.g. userID, fileType).
 No issue on the first day of test, and stopped for 3 days. I restart the test 
 on 4th day, 1 of the users failed to query the files (timeout exception 
 received). Most of the users are still okay with the query.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2647) Cassandra can't find jamm on startup

2011-05-13 Thread Eric Evans (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Evans updated CASSANDRA-2647:
--

Attachment: 
v1-0001-CASSANDRA-2647-move-jars-under-lib-to-satisfy-hard-cod.txt

 Cassandra can't find jamm on startup
 

 Key: CASSANDRA-2647
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2647
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
Affects Versions: 0.8 beta 1
 Environment: Ubuntu 10.04 Lucid
Reporter: Marcus Bointon
Assignee: paul cannon
Priority: Minor
 Fix For: 0.8.1

 Attachments: 
 v1-0001-CASSANDRA-2647-move-jars-under-lib-to-satisfy-hard-cod.txt


 I installed the Debian package (from 
 http://www.apache.org/dist/cassandra/debian unstable) of Cassandra 0.8beta2 
 on Ubuntu 10.04 with the sun jdk over a working copy of 0.7.2. It broke on 
 restart.
 On startup it gives this:
 {code}
 Error occurred during initialization of VM
 agent library failed to init: instrument
 Error opening zip file or JAR manifest missing : /lib/jamm-0.2.2.jar
 {code}
 /etc/cassandra/cassandra-env.sh contains this:
 {code}
 # add the jamm javaagent
 check_openjdk=$(java -version 21 | awk '{if (NR == 2) {print $1}}')
 if [ $check_openjdk != OpenJDK ]
 then
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.2.jar
 fi
 {code}
 By default, CASSANDRA_HOME is not set, so it's looking in /lib for this jar. 
 It seems CASSANDRA_HOME should be set to /usr/share/cassandra, since that's 
 where jamm-0.2.2.jar is installed, but that means the path is still wrong.
 I set CASSANDRA_HOME to /usr/share/cassandra and changed the JVM_OPTS line to 
 this:
 {code}
 JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/jamm-0.2.2.jar
 {code}
 and then cassandra started ok.
 Is this a bug or did I miss something?
 I also noticed that this line appears to be the only use of CASSANDRA_HOME.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2648) Failing while starting Cassandar server : Corrupt (negative) value length encountered


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033431#comment-13033431
 ] 

Stu Hood commented on CASSANDRA-2648:
-

Assuming this is random corruption, you should be able to start the node by 
moving that commitlog out of the way. If the data is not sensitive, it would 
probably be helpful if you could attach a (compressed copy).

Was there anything unusual about how the node went offline?

 Failing while starting Cassandar server : Corrupt (negative) value length 
 encountered
 -

 Key: CASSANDRA-2648
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2648
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.7.4
 Environment: Linux
Reporter: Olivier Smadja

  INFO 16:50:35,307 reading saved cache 
 /home/tapix/data/cassandra/data/saved_caches/tapix_prod-StreamPostsStatistics-KeyCache
  INFO 16:50:35,316 Opening 
 /home/tapix/data/cassandra/data/tapix_prod/StreamPostsStatistics-f-121
  INFO 16:50:35,320 Opening 
 /home/tapix/data/cassandra/data/tapix_prod/StreamPostsStatistics-f-120
  INFO 16:50:35,329 Opening 
 /home/tapix/data/cassandra/data/tapix_prod/StreamLine-f-9
  INFO 16:50:35,352 Creating new commitlog segment 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1305316235352.log
  INFO 16:50:35,362 Replaying 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1303829569725.log, 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1303948043185.log, 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1304361402015.log, 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1304728796807.log, 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1305208776962.log, 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1305316204091.log
  INFO 16:50:35,536 Finished reading 
 /home/tapix/data/cassandra/data/commitlog/CommitLog-1303829569725.log
 ERROR 16:50:35,537 Exception encountered during startup.
 java.io.IOException: Corrupt (negative) value length encountered
   at 
 org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:269)
   at 
 org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
   at 
 org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
   at 
 org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
   at 
 org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
   at 
 org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:380)
   at 
 org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:253)
   at 
 org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156)
   at 
 org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173)
   at 
 org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
   at 
 org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
 Exception encountered during startup.
 java.io.IOException: Corrupt (negative) value length encountered
   at 
 org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:269)
   at 
 org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
   at 
 org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
   at 
 org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
   at 
 org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
   at 
 org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:380)
   at 
 org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:253)
   at 
 org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156)
   at 
 org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173)
   at 
 org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
   at 
 org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-1608) Redesigned Compaction