Re: Repairing lost data

2011-08-27 Thread Anthony Molinaro
I'm pretty sure that was a bug fixed in a later 0.6.x release so you might be able to upgrade and the exceptions might go away. We run 0.6.13 with a minor mod to support data expiration and will probably do so indefinitely since there no way to upgrade without shutting our site down :(

Re: Upgrading from 0.6 to 0.7.0

2011-01-21 Thread Anthony Molinaro
-- Anthony Molinaro antho...@alumni.caltech.edu

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
Pelops. Has anyone done a similar upgrade of a live cluster? How did you go about? Is there at least a way to avoid having to upgrade both server side and client side simultaneously? Thanks, Daniel -- Anthony

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
, Jan 19, 2011 at 10:04:54PM +, Aaron Morton wrote: Your right, forgot about the change to binary keys :) Forgot what I said.  A On 20 Jan, 2011,at 11:01 AM, Anthony Molinaro antho...@alumni.caltech.edu wrote: As far as I can tell, it is impossible to run a 0.6 client against

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
Really, my bad, I though they were, but maybe I'm confusing that with protobuf, I work with too many serialization formats :(. -Anthony On Wed, Jan 19, 2011 at 04:46:48PM -0600, Jonathan Ellis wrote: On Wed, Jan 19, 2011 at 4:34 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
the clients and the server. -Anthony On Wed, Jan 19, 2011 at 05:05:02PM -0800, Anthony Molinaro wrote: Really, my bad, I though they were, but maybe I'm confusing that with protobuf, I work with too many serialization formats :(. -Anthony On Wed, Jan 19, 2011 at 04:46:48PM -0600, Jonathan Ellis

Re: Cold boot performance problems

2010-10-08 Thread Anthony Molinaro
On Fri, Oct 08, 2010 at 05:31:28PM -0700, Dave Viner wrote: Has anyone found solid step-by-step docs on how to raid0 the ephemeral disks in ec2 for use by Cassandra? No, but here's a script I used to raid0 3 ephemerals in an xlarge instance. You can edit the top part for different configs for

Data Center Move

2010-09-02 Thread Anthony Molinaro
-- Anthony Molinaro antho...@alumni.caltech.edu

Re: Data Center Move

2010-09-02 Thread Anthony Molinaro
be able to use those public addresses to migrate to some other site outside of EC2. Am I missing something obvious? (Quite possible, since I haven't actually tested this) On Thu, Sep 02, 2010 at 01:09:46PM -0500, Anthony Molinaro wrote: Hi, We're running cassandra 0.6.4, and need

Re: Cassandra HAProxy

2010-08-29 Thread Anthony Molinaro
On Sun, Aug 29, 2010 at 12:20:10PM -0700, Benjamin Black wrote: On Sun, Aug 29, 2010 at 11:04 AM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I don't know it seems to tax our setup of 39 extra large ec2 nodes, its also closer to 24000 reqs/sec at peak since there are different

Re: Cassandra HAProxy

2010-08-28 Thread Anthony Molinaro
? -- Anthony Molinaro antho...@alumni.caltech.edu

Re: what causes MESSAGE-DESERIALIZER-POOL to spike

2010-07-26 Thread Anthony Molinaro
-- Anthony Molinaro antho...@alumni.caltech.edu

Re: Bootstrap question

2010-07-21 Thread Anthony Molinaro
. Any chance you could apply the patch for 1221 and test? Gary. On Tue, Jul 20, 2010 at 16:45, Anthony Molinaro antho...@alumni.caltech.edu wrote: I see this in the old nodes DEBUG [WRITE-/10.220.198.15] 2010-07-20 21:15:50,366 OutboundTcpConnection.java (line 142) attempting to connect

Re: Bootstrap question

2010-07-20 Thread Anthony Molinaro
the system keyspace first, just in case. On Sun, Jul 18, 2010 at 2:01 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: Yeah, I tried all that already and it didn't seem to work, no new nodes will bootstrap, which makes me think there's some saved state somewhere, preventing a new node

Re: Bootstrap question

2010-07-18 Thread Anthony Molinaro
for me. 1. Restart the bootstraping node 2. If I see streaming 0/ I restart the node and all the streaming nodes 3. Restart all the nodes 4. If there is data in the bootstraing node I delete it before I restart. Good luck Shimi On Sun, Jul 18, 2010 at 12:21 AM, Anthony Molinaro antho

Bad SSTable killing compaction?

2010-07-17 Thread Anthony Molinaro
-- Anthony Molinaro antho...@alumni.caltech.edu

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
to bootstrap to. Who owns the ranges that contain those tokens? On Wed, Jul 14, 2010 at 5:58 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: Hi,  I have a 0.6.3 cluster which contains 6 nodes.  I added 6 new nodes by setting AutoBootstrap to true and setting an InitialToken

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
Oh, and looking at the load on the new machines it appears that New 2 and New 6 have gotten some data (although neither is in the ring yet). Not sure if that clears anything up though. -Anthony On Thu, Jul 15, 2010 at 01:28:06PM -0700, Anthony Molinaro wrote: This is a cluster which

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
, 2010 at 03:43:49PM -0500, Jonathan Ellis wrote: On Thu, Jul 15, 2010 at 3:28 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: Is the fact that 2 new nodes are in the range messing it up? Probably.  And if so how do I recover (I'm thinking, shutdown new nodes 2,3,4,5

Bootstrap question

2010-07-14 Thread Anthony Molinaro
be looking at? Thanks, -Anthony -- Anthony Molinaro antho...@alumni.caltech.edu

Re: Cassandra on AWS across Regions

2010-06-29 Thread Anthony Molinaro
(US-EAST), cuz they can communicate with private IP. Can any one share What is the recommended approach here? Lenin -- twitter: leningali skype: galilenin Cell:513.382.3371 -- Anthony Molinaro

JVM Options for Production

2010-06-14 Thread Anthony Molinaro
? Thanks, -Anthony -- Anthony Molinaro antho...@alumni.caltech.edu

Re: GMFD messages

2010-05-27 Thread Anthony Molinaro
MessageDeserializer had that message). Do gossip messages flow through the MessageDeserializer? Thanks for the response, -Anthony On Tue, May 25, 2010 at 5:33 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: Hi,  I just noticed I have lots of these messages INFO [GMFD:1] 2010

GMFD messages

2010-05-25 Thread Anthony Molinaro
be causing this sort of thing? This cluster is now at 27 m1.xlarge boxes on ec2 running 0.6.2 of some flavor. Thanks, -Anthony -- Anthony Molinaro antho...@alumni.caltech.edu

Compaction JMX Stats?

2010-05-20 Thread Anthony Molinaro
-- Anthony Molinaro antho...@alumni.caltech.edu

Re: Overfull node

2010-05-17 Thread Anthony Molinaro
, -- Anthony Molinaro antho...@alumni.caltech.edu

Re: Error in TBaseHelper compareTo(byte [] a , byte [] b)

2010-05-03 Thread Anthony Molinaro
-- Anthony Molinaro antho...@alumni.caltech.edu

Re: Batch mutate doesn't work

2010-04-30 Thread Anthony Molinaro
, value = value, timestamp = 2 } } } ] } }, Try that, -Anthony -- Anthony Molinaro antho...@alumni.caltech.edu

Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
down. The seed node has all nodes. Anyone seen this? How can I get those 4 nodes to see the missing node? If a known issue has it been fixed in 0.6 or newer? Thanks, -Anthony -- Anthony Molinaro

Re: Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
On Fri, Apr 23, 2010 at 12:41:17PM -0500, Jonathan Ellis wrote: On Fri, Apr 23, 2010 at 12:30 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: Some nodes appear in the ring from some nodes, but not others.  Right now I have 14 nodes, 10 of those nodes have the same output

Re: Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
On Fri, Apr 23, 2010 at 01:17:21PM -0500, Jonathan Ellis wrote: On Fri, Apr 23, 2010 at 1:12 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I'm not sure how it would get this, maybe I need to restart my seed node? It's worth a try. Sounds like you found an unusual bug in gossip

Re: Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
:11PM -0700, Anthony Molinaro wrote: On Fri, Apr 23, 2010 at 01:17:21PM -0500, Jonathan Ellis wrote: On Fri, Apr 23, 2010 at 1:12 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I'm not sure how it would get this, maybe I need to restart my seed node? It's worth a try. Sounds

Re: Clarification on Ring operations in Cassandra 0.5.1

2010-04-21 Thread Anthony Molinaro
the sstables while the system was running (since they shouldn't be changing). Then in quick succession removetoken and bootstrap with the old token. Probably grasping at straws here :b Thanks for the answers, -Anthony On Wed, Apr 21, 2010 at 11:02 AM, Anthony Molinaro antho

Re: Cassandra 0.5.1 restarts slow

2010-04-21 Thread Anthony Molinaro
at 4:57 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: Hi,  I have a cassandra cluster where a couple things are happening.  Every once in a while a node will start to get backed up.  Checking tpstats I see a very large value for ROW-MUTATION-STAGE.  Sometimes it will be able

Re: Cassandra 0.5.1 restarts slow

2010-04-21 Thread Anthony Molinaro
On Wed, Apr 21, 2010 at 12:52:32PM -0500, Jonathan Ellis wrote: On Wed, Apr 21, 2010 at 12:45 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: as for why it backs up in the first place before the restart, you can either (a) throttle writes [set your timeout lower, make your clients

Re: Clarification on Ring operations in Cassandra 0.5.1

2010-04-21 Thread Anthony Molinaro
On Wed, Apr 21, 2010 at 12:05:07PM -0500, Jonathan Ellis wrote: On Wed, Apr 21, 2010 at 11:31 AM, Anthony Molinaro antho...@alumni.caltech.edu wrote: On Wed, Apr 21, 2010 at 11:08:19AM -0500, Jonathan Ellis wrote: Yes, that looks right, where token really close means slightly less than

Re: Cassandra 0.5.1 restarts slow

2010-04-21 Thread Anthony Molinaro
On Wed, Apr 21, 2010 at 01:24:45PM -0500, Jonathan Ellis wrote: On Wed, Apr 21, 2010 at 1:11 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: Interesting, in the config I see  !-- Time to wait for a reply from other nodes before failing the command --  RpcTimeoutInMillis5000

Clarification on Ring operations in Cassandra 0.5.1

2010-04-15 Thread Anthony Molinaro
= false, what happens? Does it join the ring but without data and without token range? Can I then 'nodeprobe move token for range I want to take over', and achieve the same as step 2 above? Thanks, -Anthony -- Anthony Molinaro

Re: Recovery from botched compaction

2010-04-13 Thread Anthony Molinaro
On Tue, Apr 13, 2010 at 10:54:51AM -0500, Jonathan Ellis wrote: On Sat, Apr 10, 2010 at 2:24 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote:  This is sort of a pre-emptive question as the compaction I'm doing hasn't failed yet but I expect it to any time now.  I have a cluster which