Re: Nodes Dying in 2.1.2

2015-01-03 Thread Phil Burress
tickets. =Rob On Mon, Dec 29, 2014 at 1:40 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Dec 24, 2014 at 9:41 AM, Phil Burress philtburr...@gmail.com wrote: Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying. The kernel is killing the process due to out of memory

Re: 答复: Downgrade from 2.1.2 to 2.1.1

2014-12-31 Thread Phil Burress
:18600360358 地址:北京市大兴区 朝林广场25层 邮编:100101 = 网购上京东,省钱又放心! www.jd.com *发件人:* Phil Burress [mailto:philtburr...@gmail.com] *发送时间:* 2014年12月31日 2:53 *收件人:* user@cassandra.apache.org *主题:* Re: Downgrade from 2.1.2 to 2.1.1 Thanks Rob. On Tue

Re: Downgrade from 2.1.2 to 2.1.1

2014-12-30 Thread Phil Burress
Thanks Rob. On Tue, Dec 30, 2014 at 1:38 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Dec 30, 2014 at 9:42 AM, Phil Burress philtburr...@gmail.com wrote: We are having a lot of problems with release 2.1.2. It was suggested here we should downgrade to 2.1.1 if possible

Nodes Dying in 2.1.2

2014-12-24 Thread Phil Burress
Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying. The kernel is killing the process due to out of memory: kernel: Out of memory: Kill process 6267 (java) score 998 or sacrifice child Appears to only occur during compactions. We've tried playing with the heap settings but

Re: nodetool repair -snapshot option?

2014-07-01 Thread Phil Burress
://github.com/hancockks/cassandra_range_repair On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress philburress...@gmail.com wrote: @Paulo, this is very cool! Thanks very much for the link! On Mon, Jun 30, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes paulo.mo...@chaordicsystems.com wrote: If you

nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
We are running into an issue with nodetool repair. One or more of our nodes will die with OOM errors when running nodetool repair on a single node. Was reading this http://www.datastax.com/dev/blog/advanced-repair-techniques and it mentioned using the -snapshot option, however, that doesn't appear

Re: nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
We are running repair -pr. We've tried subrange manually and that seems to work ok. I guess we'll go with that going forward. Thanks for all the info! On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia chovatia.jayd...@gmail.com wrote: Are you running full repair or on subset? If you are

Re: nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
One last question. Any tips on scripting a subrange repair? On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress philburress...@gmail.com wrote: We are running repair -pr. We've tried subrange manually and that seems to work ok. I guess we'll go with that going forward. Thanks for all the info

Re: nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
:26 PM, Phil Burress philburress...@gmail.com wrote: One last question. Any tips on scripting a subrange repair? On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress philburress...@gmail.com wrote: We are running repair -pr. We've tried subrange manually and that seems to work ok. I guess we'll go

Ec2 Network I/O

2014-05-19 Thread Phil Burress
Has anyone experienced network i/o issues with ec2? We are seeing a lot of these in our logs: HintedHandOffManager.java (line 477) Timed out replaying hints to /10.0.x.xxx; aborting (15 delivered) and these... Cannot handshake version with /10.0.x.xxx and these... java.io.IOException: Cannot

Re: Recommended Approach for Config Changes

2014-04-28 Thread Phil Burress
Hobbs ty...@datastax.com wrote: On Fri, Apr 25, 2014 at 10:43 AM, Phil Burress philburress...@gmail.comwrote: Thanks. I made a change to a single node and it took almost an hour to rejoin the cluster (go from DN to UP in nodetool status). The cluster is pretty much idle right now and has

Recommended Approach for Config Changes

2014-04-25 Thread Phil Burress
If I wanted to make a configuration change to a single node in a cluster, what is the recommended approach for doing that? Is it ok to just stop that instance, make the change and then restart it?

Re: Bootstrap Timing

2014-04-25 Thread Phil Burress
Just a follow-up on this for any interested parties. Ultimately we've determined that the bootstrap/join process is broken in Cassandra. We ended up creating an entirely new cluster and migrating the data. On Mon, Apr 21, 2014 at 10:32 AM, Phil Burress philburress...@gmail.comwrote: The new

Re: Bootstrap Timing

2014-04-25 Thread Phil Burress
Cassandra 2.0.6 On Fri, Apr 25, 2014 at 10:31 AM, James Rothering jrother...@codojo.mewrote: What version of C* is this? On Fri, Apr 25, 2014 at 6:55 AM, Phil Burress philburress...@gmail.comwrote: Just a follow-up on this for any interested parties. Ultimately we've determined

Re: Recommended Approach for Config Changes

2014-04-25 Thread Phil Burress
: Yes. Some changes you can manually have take affect without a restart (ie compactionthroughput, things settable from jmx). There is also config changes you cant really make like switching the snitch and such without a big todo. --- Chris On Apr 25, 2014, at 8:53 AM, Phil Burress

Re: Bootstrap Timing

2014-04-21 Thread Phil Burress
might be able to give you a better idea. Steve On Fri, Apr 18, 2014 at 1:43 PM, Phil Burress philburress...@gmail.comwrote: First, I just stopped 2 of the nodes and left one running. But this morning, I stopped that third node, cleared out the data, restarted and let it rejoin again

Re: Bootstrap Timing

2014-04-19 Thread Phil Burress
On Fri, Apr 18, 2014 at 1:43 PM, Phil Burress philburress...@gmail.comwrote: First, I just stopped 2 of the nodes and left one running. But this morning, I stopped that third node, cleared out the data, restarted and let it rejoin again. It appears streaming is done (according to netstats

Re: Bootstrap Timing

2014-04-18 Thread Phil Burress
PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Apr 16, 2014 at 1:56 PM, Phil Burress philburress...@gmail.comwrote: I've shut down two of the nodes and am bootstrapping one right now. Is there any way to tell when it will finish bootstrapping? nodetool netstats will show the progress

Re: Bootstrap Timing

2014-04-18 Thread Phil Burress
, 2014 at 10:40 AM, Robert Coli rc...@eventbrite.comwrote: On Fri, Apr 18, 2014 at 5:05 AM, Phil Burress philburress...@gmail.comwrote: nodetool netstats shows 84 files. They are all at 100%. Nothing showing in Pending or Active for Read Repair Stats. I'm assuming this means it's done

Bootstrap Timing

2014-04-16 Thread Phil Burress
Greetings, How long does bootstrapping typically take? I have 3 existing nodes in our cluster with about 40GB each. I've added three new nodes to the cluster. They have been in bootstrap mode for a little over 3 days now. Should I be concerned? Is there a way to tell how long it will take to

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
Thanks very much for the response. I'm not using vnodes, does that matter? On Wed, Apr 16, 2014 at 2:13 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Apr 16, 2014 at 11:10 AM, Phil Burress philburress...@gmail.comwrote: How long does bootstrapping typically take? I have 3 existing

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
Also, one more quick question. For the new nodes, do I add all three existing nodes as seeds? Or just add one? On Wed, Apr 16, 2014 at 2:16 PM, Phil Burress philburress...@gmail.comwrote: Thanks very much for the response. I'm not using vnodes, does that matter? On Wed, Apr 16, 2014 at 2:13

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
Thanks! On Wed, Apr 16, 2014 at 2:50 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Apr 16, 2014 at 11:16 AM, Phil Burress philburress...@gmail.comwrote: Thanks very much for the response. I'm not using vnodes, does that matter? Not in your case. In some cases it is safe

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
I've shut down two of the nodes and am bootstrapping one right now. Is there any way to tell when it will finish bootstrapping? On Wed, Apr 16, 2014 at 2:56 PM, Phil Burress philburress...@gmail.comwrote: Thanks! On Wed, Apr 16, 2014 at 2:50 PM, Robert Coli rc...@eventbrite.com wrote