I'm looking for some guidance on how to model some stat tracking over time,
bucketed to some type of interval (15 min, hour, etc).
As an example, let's say I would like to track network traffic throughput and
bucket it to 15 minute intervals. In our old model, using thrift I would
create a
the
responses from all nodes in other DC will go over WAN.
On Thu, Oct 25, 2012 at 10:44 AM, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf...@azaleos.com wrote:
We have a 5 node cluster, with a matching 5 nodes for DR in another data
center. With a replication factor of 3, does
We have a 5 node cluster, with a matching 5 nodes for DR in another data
center. With a replication factor of 3, does the node I send a write too
attempt to send it to the 3 servers in the DR also? Or does it send it to 1
and let it replicate locally in the DR environment to save bandwidth
Is there anything I can do on the configuration side to prevent nodes from
going OOM due to queries that will read large amounts of data and exceed the
heap available?
For the past few days of we had some nodes consistently freezing/crashing with
OOM. We got a heap dump into MAT and figured
if it would work.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/08/2012, at 9:10 AM, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf...@azaleos.com wrote:
I believe what may be really going on is that my schema is in a bad
cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf...@azaleos.com wrote:
Same results. I restarted the node also to see if it just wasn't picking up
the changes
in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf...@azaleos.com wrote:
Same results. I restarted the node also to see if it just wasn't picking up
the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using
for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/08/2012, at 6:53 PM, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf
, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf...@azaleos.com wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2
, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey
bryce.godf
-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf...@azaleos.com wrote:
We are in the process of building out a new DR system in another Data Center,
and we want to mirror our Cassandra environment to that DR. I have a couple
:
http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey
bryce.godf...@azaleos.commailto:bryce.godf...@azaleos.com wrote:
We are in the process of building out a new DR system in another
We are in the process of building out a new DR system in another Data Center,
and we want to mirror our Cassandra environment to that DR. I have a couple
questions on the best way to do this after reading the documentation on the
Datastax website. We didn't initially plan for this to be a DR
What is the process for joining a new data center to an existing cluster as DR?
We have a 5 node cluster in our primary DC, and want to bring up 5 more in our
2nd data center purely for DR. How should these new nodes be joined to the
cluster and be seen as the 2nd data center? Do the new
This may not be directly related to the upgrade to 1.1.2, but I was running on
1.1.0 for a while with no issues, and I did the upgrade to 1.1.2 a few days ago.
2 of my nodes started throwing lots of promote exceptions, and then a lot of
the beforeAppend exceptions from then on every few
.
On 16.07.2012, at 20:32, Bryce Godfrey wrote:
This may not be directly related to the upgrade to 1.1.2, but I was running on
1.1.0 for a while with no issues, and I did the upgrade to 1.1.2 a few days ago.
2 of my nodes started throwing lots of promote exceptions, and then a lot of
the beforeAppend
at 12:28 AM, Bryce Godfrey bryce.godf...@azaleos.com
wrote:
As the new node starts up I get this error before boostrap starts:
INFO 08:20:51,584 Enqueuing flush of
Memtable-schema_columns@1493418651(0/0
serialized/live bytes, 1 ops)
INFO 08:20:51,584 Writing Memtable-schema_columns
As the new node starts up I get this error before boostrap starts:
INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0
serialized/live bytes, 1 ops)
INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
serialized/live bytes, 1 ops)
INFO 08:20:51,589 Completed
I'll try to get some log files for this with DEBUG enabled. Tough on
production though.
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Monday, June 04, 2012 11:15 AM
To: user@cassandra.apache.org
Subject: Re: 1.1 not removing commit log files?
Apply the local hint mutation follows
not reclaim the space either and delete any log files.
Bryce Godfrey | Sr. Software Engineer | Azaleos
Corporationhttp://www.azaleos.com/
From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]
Sent: Tuesday, May 22, 2012 1:10 PM
To: user@cassandra.apache.org
Subject: RE: 1.1 not removing commit
the internal hard coded default for commitlog_total_space_in_mb
If you are seeing more that 4GB of commit log files let us know.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote:
Thanks, I'll give
The commit log drives on my nodes keep slowly filling up. I don't see any
errors in my logs that are indicating any issues that I can map to this issue.
Is this how 1.1 is supposed to work now? Previous versions seemed to keep this
drive at a minimum as it flushed.
/dev/mapper/mpathf 25G
This is the second node I've joined to my cluster in the last few days, and so
far both have become stuck at 100% on a large file according to netstats. This
is on 1.0.9, is there anything I can do to make it move on besides restarting
Cassandra? I don't see any errors or warns in logs for
Per CF or per Row TTL would be very usefull for me also with our timeseries
data.
-Original Message-
From: Igor [mailto:i...@4friends.od.ua]
Sent: Wednesday, April 18, 2012 6:06 AM
To: user@cassandra.apache.org
Subject: Re: size tiered compaction - improvement
For my use case it would
On Mon, Apr 16, 2012 at 10:45 PM, Bryce Godfrey bryce.godf...@azaleos.com
wrote:
I keep running into this with my testing (on a windows box), Is this just a
OOM for RAM?
How much RAM do you have? Do you use completely standard settings? Do you also
OOM if you try the same test with Cassandra
I keep running into this with my testing (on a windows box), Is this just a OOM
for RAM?
ERROR [COMMIT-LOG-ALLOCATOR] 2012-04-16 13:36:18,790
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Map failed
://www.thelastpickle.com
On 16/03/2012, at 12:58 PM, Bryce Godfrey wrote:
We were having some occasional memory pressure issues, but we just added some
more RAM a few days ago to the nodes and things are running more smoothly now,
but in general nodes have not been going up and down.
I tried
The system HintsColumnFamily seems large in my cluster, and I want to track
down why that is. I try invoking listEndpointsPendingHints() for
o.a.c.db.HintedHandoffManager and it never returns, and also freezes the node
that its invoked against. It's a 3 node cluster, and all nodes have been
Forgot to mention that this is on 1.0.8
From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]
Sent: Wednesday, March 14, 2012 12:34 PM
To: user@cassandra.apache.org
Subject: Large hints column family
The system HintsColumnFamily seems large in my cluster, and I want to track
down why
I'm seeing this also, and my nodes have started crashing with too many open
file errors. Running lsof I see lots of these open tmp files.
java 8185 root 911u REG 8,32 38
129108266
)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
Now I see no streams going on between any nodes, and the node is still listed
as moving when viewing the ring.
From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com]
Sent: Sunday, December 11, 2011 11:02 PM
To: user
I have a dead node I need to remove from the cluster so that I can rebalance
among the existing servers (can't replace it for a while).
I used nodetool removetoken and it's been stuck in the leaving state for over
a day now. I've tried a rolling restart, which kicks of some streaming for a
I'm seeing this same problem after upgrade to 1.0.3 from .8
Nothing changed with the column family storing the counters, but now it just
constantly times out trying to increment them. No errors in the event logs or
any other issues with my cluster.
Did you find a resolution?
From: Carlos
to 1.0.1
One possibility: If you're overloading the cluster, replicas will drop updates
to avoid OOMing. (This is logged at WARN level.) Before 1.x Cassandra would
just let that slide, but with w/ 1.0 it started recording hints for those.
On Thu, Nov 3, 2011 at 7:17 PM, Bryce Godfrey bryce.godf
)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Bryce Godfrey | Sr. Software Engineer | Azaleos
Corporationhttp://www.azaleos.com/ | T: 206.926.1978 | M: 206.849.2477
Nope. I did alter two of my own column families to use Leveled compaction and
then ran scrub on each node, is the only change I have made from the upgrade.
Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation | T: 206.926.1978 |
M: 206.849.2477
-Original Message-
From: Jonathan
Disk utilization is actually about 80% higher than what is reported for
nodetool ring across all my nodes on the data drive
Bryce Godfrey | Sr. Software Engineer | Azaleos
Corporationhttp://www.azaleos.com/ | T: 206.926.1978 | M: 206.849.2477
From: Dan Hendry [mailto:dan.hendry.j...@gmail.com
:52 AM
To: user@cassandra.apache.org
Subject: Re: Problem after upgrade to 1.0.1
Does restarting the node fix this?
On Thu, Nov 3, 2011 at 1:51 PM, Bryce Godfrey bryce.godf...@azaleos.com wrote:
Disk utilization is actually about 80% higher than what is reported
for nodetool ring across all my
@cassandra.apache.org
Subject: Re: Problem after upgrade to 1.0.1
I found the problem and posted a patch on
https://issues.apache.org/jira/browse/CASSANDRA-3451. If you build with that
patch and rerun scrub the exception should go away.
On Thu, Nov 3, 2011 at 2:08 PM, Bryce Godfrey bryce.godf...@azaleos.com
I'm wondering what the consensus is for running a Cassandra cluster on top of
Windows boxes? We are currently running a small 5 node cluster on top of
CentOS without problems, so I have no desire to move. But we are a windows
shop, and I have an IT department that is scared of Linux since
Taking the cluster down completely did remove the phantom node. The
hintscolumnfamily is causing a lot of commit logs to back up and threaten the
commit log drive to run out of space. A manual flush of that column family
always clears out the files though.
-Original Message-
From:
Row cache: disabled
Compacted row minimum size: 2816160
Compacted row maximum size: 386857368
Compacted row mean size: 120432714
Is there a way for me to manually remove this dead node?
-Original Message-
From: Bryce Godfrey
and 20.3 think 20.1 is still around ?
Cheers
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 20/08/2011, at 9:48 AM, Bryce Godfrey wrote:
I'm on 0.8.4
I have removed a dead node from the cluster using nodetool removetoken
command
Developer
@aaronmorton
http://www.thelastpickle.com
On 22/08/2011, at 6:23 AM, Bryce Godfrey wrote:
Both .2 and .3 list the same from the mbean that Unreachable is empty
collection, and Live node lists all 3 nodes still:
192.168.20.2
192.168.20.3
192.168.20.1
The removetoken was done a few
I'm on 0.8.4
I have removed a dead node from the cluster using nodetool removetoken command,
and moved one of the remaining nodes to rebalance the tokens. Everything looks
fine when I run nodetool ring now, as it only lists the remaining 2 nodes and
they both look fine, owning 50% of the
-server.properties.
maki
2011/6/1 Bryce Godfrey bryce.godf...@azaleos.com:
Hi there, I'm a bit new to Linux and Cassandra so I'm hoping someone
can help me with this.
I've been evaluating Cassandra for the last few days and I'm now
having a problem starting up the service. I receive this error
46 matches
Mail list logo
