http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/ - Gives some
background information (specific to 0.8 but still valid for 1.0 I believe).
Not quite sure why a warning message is logged but a ration of 1 may occur
for column families with a very high update to insert ratio.
Dan
://issues.apache.org/jira/browse/CASSANDRA-3592
Dan Hendry
(403) 660-2297
@cassandra.apache.org
Subject: Re: Slow Compactions - CASSANDRA-3592
Does your issue look similar this one?
https://issues.apache.org/jira/browse/CASSANDRA-3532
It is also dealing with compactaion taking 10X longer in 1.0.X
On 12/13/2011 09:00 AM, Dan Hendry wrote:
I have been observing that major
19:42
To: user@cassandra.apache.org
Subject: Re: split large sstable
Dne 17.11.2011 17:42, Dan Hendry napsal(a):
What do you mean by ' better file offset caching'? Presumably you mean
'better page cache hit rate'?
fs metadata used to find blocks in smaller files are cached better.
Large files
I they are not limited to repeating values but the Datastax docs[1] on
secondary indexes certainly seem to indicate they would be a poor fit for this
case (high read load, many unique values).
[1] http://www.datastax.com/docs/1.0/ddl/indexes
Dan
From: Maciej Miklas
My understanding is that support for super columns will remain within the
thrift API for the foreseeable future (perhaps indefinitely?) but under the
covers, super column families will get transitioned to regular column
families with composite column names (
What do you mean by ' better file offset caching'? Presumably you mean
'better page cache hit rate'? Out of curiosity, why do you think this? What
data are you seeing which makes you think it's better? I am certainly not
even close to a virtual memory or page caching expert but I am pretty sure
Your first approach, skinny rows, will almost certainly be a better solution
although it never hurts to experiment for yourself. Even for low end hardware
(for sake of argument, EC2 m1.smalls), a few million rows is basically nothing
(again though, I encourage you to verify for yourself). For
Last night one of my nodes died inexplicably, with no log entries anywhere
indicating the reason (cassandras log, /var/log/messages, etc). I tried to
restart it but the node will not restart as it fails with the errors below
when replaying the commit log. I should point out that the cluster is
I really don't recommend using t1.micros. The problem with them is that they
have CPU bursting, basically meaning you get lots of CPU resources for a
short time but if you use more than you have been allocated you get
basically nothing for 10+ seconds afterwards. By 'basically nothing' I
really
seems to be behaving normally.
Thoughts?
Dan Hendry
Active Pending Completed Blocked All
time blocked
MemtablePostFlusher 118 16 0
0
Dan
From: Dan Hendry [mailto:dan.hendry.j...@gmail.com]
Sent: November-10-11 14:49
To: 'user@cassandra.apache.org'
Subject: 1.0.2 Assertion Error
Just
the MemtablePostFlusher is unfortunately
related. Restarting the
node would fix but we need to make that more solid too.
--
Sylvain
On Thu, Nov 10, 2011 at 9:04 PM, Dan Hendry dan.hendry.j...@gmail.com
wrote:
Just happened again, seems to be with the same column family (at least on
a
flusher thread
I *thought* Cassandra was supposed to have a crash only design[1]. My
understanding is that it is safe to simply kill the process and with the
regular TERM signal and, shutdown would be blocked on fsyncing the commit log
but nothing else (obviously not true if you kill -9 the sucker) even when
Regarding load growth, presumably you are referring to the load as reported
by JMX/nodetool. Have you actually looked at the disk utilization on the
nodes themselves? Potential issue I have seen:
http://www.mail-archive.com/user@cassandra.apache.org/msg18142.html
Dan
From: Bryce Godfrey
Uh, so look at your await time of *107.3*. From the iostat man page: await:
The average time (in milliseconds) for I/O requests issued to the device to
be served. This includes the time spent by the requests in queue and the
time spent servicing them.
If the key you are reading from is not
of 2-300ms for getting a single row. This seems slow, but is it unusual?
What are those numbers? 2 ms being average? 300 ms a 95th/99th percentile? A
value you saw once? Yes, this *seems* slow given your row definition but
without knowing what the value represents it's almost impossible to
2. ... So I am going to use rotational disk for the commit log and an SSD
for data. Does this make sense?
Yes, just keep in mind however that the primary characteristic of SSDs is
lower seek times which translates into faster random access. We have a
similar Cassandra use case (time series
. Is this a known issue or has anybody else observed a similar pattern?
Dan Hendry
(403) 660-2297
The R+W RF requirement for strong consistency applies regardless of
whether your data is 'immutable' or is being updated. A W=1, R=1 approach
will not guarantee consistency between reads and writes.
R=1 might cassandra look on one of the two nodes, find no data there, and
prematurely give
Have you installed 'jna'? On RHEL (6 at least) it should be possible using
the default yum repos. You need the native code and the JAR in Cassandras
classpath from what I understand.
Dan
From: eczec...@gmail.com [mailto:eczec...@gmail.com] On Behalf Of Eric Czech
Sent: September-01-11
86GB in commitlog and 42GB in data
Whoa, that seems really wrong, particularly given your data spans 13 months.
Have you changed any of the default cassandra.yaml setting? What is the
maximum memtable_flush_after across all your CFs? Any warnings/errors in the
Cassandra log?
Out of
First off, what version of Cassandra are you using?
We've noticed that when we restart cassandra disk utilization decreases
dramatically
Presumably you mean 'utilization' as in free space. Specifically on a
restart, this type of behavior is likely due to Cassandra deleting compacted
SSTables.
on this machine and no
repairs running across the cluster
. Hinted handoff is disabled
Any insight would be appreciated.
Dan Hendry
: Memtable flush thresholds - what am I missing?
See http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/,
specifically the section on memtable_total_space_in_mb
On Thu, Aug 18, 2011 at 2:43 PM, Dan Hendry dan.hendry.j...@gmail.com
wrote:
I am in the process of trying to tune the memtable
Moving nodes does not result in downtime provide you use proper replication
factors and read/write consistencies. The typical recommendation is RF=3 and
QUORUM reads/writes.
Dan
From: Paul Loy [mailto:ketera...@gmail.com]
Sent: July-04-11 5:59
To: user@cassandra.apache.org
Subject: Re:
I think this would add a lot of complexity behind the scenes and be
conceptually confusing, particularly for new users. The Cassandra consistency
model is pretty elegant and this type of approach breaks that elegance in many
ways. It would also only really be useful when the value has a high
How would your solution deal with complete network partitions? A node being
'down' does not actually mean it is dead, just that it is unreachable from
whatever is making the decision to mark it 'down'.
Following from Ryan's example, consider nodes A, B, and C but within a
fully partitioned
a...@dude.podzone.net wrote:
On 6/16/2011 7:56 PM, Dan Hendry wrote:
How would your solution deal with complete network partitions? A node
being 'down' does not actually mean it is dead, just that it is unreachable
from whatever is making the decision to mark it 'down'.
Following from Ryan's example
Try running nodetool scrub on the cf: its pretty good at detecting and
fixing most corruption problems.
Dan
-Original Message-
From: Jonathan Colby [mailto:jonathan.co...@gmail.com]
Sent: April-15-11 15:41
To: user@cassandra.apache.org
Subject: recurring EOFException exception in 0.7.4
So Cassandra does not use an atomic commit protocol at the cluster level.
Strong consistency on a quorum read is only guaranteed *after* a successful
quorum write. The behaviour you are seeing is possible if you are reading in
the middle of a write or the write failed (which should be reported to
Uh... don't create a column family per user. Column families are meant to be
fairly static; conceptually equivalent to a table in a relational database.
Why do you need (or even want) a CF per user? Reconsider your data model, a
single column family with an inverted index for a 'user' column is
There are two layers of settings, the default, cluster wide, settings part
of the schema and exposed/modifiable via the cli and individual settings
exposed/modifiable via JMX and nodetool. Using nodetool, you are only
modifying the in memory settings for a single node, changes to those
settings
I too have seen the out of sequence response problem. My solution has just been
to retry and it seems to work. None of my mutations are THAT large ( 200
columns).
The only related information I could find points to a thrift/ubuntu bug of some
kind
Try turning on GC logging in Cassandra-env.sh, specifically:
-XX:+PrintGCApplicationStoppedTime
-Xloggc:/var/log/cassandra/gc.log
Look for things like: Total time for which application threads were
stopped: 52.8795600 seconds. Anything over about a few seconds may be
causing your
I have been having plenty of problems (on 0.7.0,
http://www.mail-archive.com/user@cassandra.apache.org/msg09341.html,
http://www.mail-archive.com/user@cassandra.apache.org/msg09230.html,
http://www.mail-archive.com/user@cassandra.apache.org/msg09122.html,
1) If I insert a key and want to verify which node it went to then how do
I
do that?
I don't think you can and there should be no reason to care. Cassandra
abstracts where data is being stored, think in terms of consistency levels
not actual nodes.
2) How can I verify if the replication is
Are you using a higher level client (hector/pelops/pycassa/etc) or the
actual thrift API? Higher level clients often pool connections and two
subsequent operations (read then write) may be performed with connections to
different nodes.
If you are sure you are using the same connection (the
handle the current issue in application but I'm sure that I will not
be able to handle some future situation in application.
So the suggestion is to use at least 3 nodes with RF=3 and CL.QUORUM for
both write and reads where high consistency is required, right?
Thanks!
2011/2/12 Dan Hendry
)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetche
r.getNextBlock(IndexedSliceReader.java:180)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(Indexe
dSliceReader.java:119)
... 22 more
Dan Hendry
(403) 660-2297
...@gmail.com]
Sent: January-24-11 19:19
To: user@cassandra.apache.org
Subject: Re: Repair on single CF not working (0.7)
On Mon, Jan 24, 2011 at 4:15 PM, Dan Hendry dan.hendry.j...@gmail.com
wrote:
I am trying to repair a single CF using nodetool. It seems like the request
to limit the repair to one
I am having yet another issue on one of my Cassandra nodes. Last night, one
of my nodes ran out of memory and crashed after flooding the logs with the
same type of errors I am seeing below. After restarting, they are popping up
again. My solution has been to drop the consistency from ALL to ONE
When this has happened to me, restarting the node you are trying to
move works. I can't remeber the exact conditions but I have also hade
to restart all nodes in the cluster simultaneously once or twice as
well.
I would love to know if there is a better way of doing it.
On Wednesday, January 26,
with this? More joy, less joy or a continuation of the
current level of joy?
Aaron
On 24/01/2011, at 9:38 AM, Dan Hendry dan.hendry.j...@gmail.com wrote:
I have run into a strange problem and was hoping for suggestions on how to fix
it (0.7.0). When compaction occurs on one node
0905,58066315307)
progress=18898300928/28535694402 - 66% from
org.apache.cassandra.streaming.StreamInSession@8df7e0c failed: requesting a
retry.
Dan Hendry
(403) 660-2297
at
org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java
:52)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdenti
tyIterator.java:69)
... 20 more
Dan Hendry
(403) 660-2297
I am having some reliability problems in my Cassandra cluster which I am
almost certain is due to GC. I was about to start delving into the guts of
the problem by turning on GC logging but I have never done any serious java
GC tuning before (time to learn I guess). As a first step however, I was
Thanks for all the info, I think I have been able to sort out my issue. The
new settings I am using are:
-Xmn512M (Very important I think)
-XX:SurvivorRatio=5 (Not very important I think)
-XX:MaxTenuringThreshold=5
-XX:ParallelGCThreads=8
-XX:CMSInitiatingOccupancyFraction=75
Since applying
On Thu, Dec 23, 2010 at 4:35 PM, Dan Hendry dan.hendry.j...@gmail.comwrote:
Your details are rather vague, what do you mean by killed? Is the
Cassandra java process still running? Any other warning or error log
messages (from either node)? Could you provide the last few Cassandra log
lines from
cassandra and start off with a fully
fresh copy?
Thanks
Alex
On Fri, Dec 24, 2010 at 1:42 PM, Dan Hendry dan.hendry.j...@gmail.com
wrote:
Hum, very strange.
More what I was trying to get at was: did the process truly die or was it
just non-responsive and looking like it was dead? It would
The main diagnosing feature of the problem I was seeing is very high system
CPU with no user CPU utilization(check with top or sar -u), vmstat showing
one process waiting for run-time but never seeming to get it, a high page
scan rate, and no Cassandra error messages (although nodes dying did
I have been having severe and strange reliability problems within my
Cassandra cluster. This weekend, all four of my nodes were down at once.
Even now I am loosing one every few hours. I have attached output from all
the system monitoring commands I can think of.
What seems to happen is that the
of my own application (java running over linux box) i tried that command to
see what the application was doing or trying to.
Kani
On Mon, Dec 20, 2010 at 3:48 PM, Dan Hendry dan.hendry.j...@gmail.com wrote:
I have been having severe and strange reliability problems within my Cassandra
by my misguided tinkering
and/or other Cassandra bugs. This time around, I have done very little with
JMX/CLI/nodetool and I can find no related Cassandra bugs.
Help/suggestions?
Dan Hendry
(403) 660-2297
Reports for the node which is down and should not be (192.168.4.17)
Report
command looks like on both *.19 and *.17
after the decommission is run. I assume you were running the ring command
on another node? I'll look into the logs more and see if anything jumps
out.
On Wed, Dec 15, 2010 at 6:37 AM, Dan Hendry dan.hendry.j...@gmail.com
wrote:
I am seeing very strange
) and am getting wildly different estimates
from different nodes. For example, one says ~50 million and another ~75
million.
Dan Hendry
(403) 660-2297
Or you can just start at the 1 + nth id given ids must be unique (you don't
have to specify an existing id as the start of a slice). You don't HAVE to
load the n + 1 record.
This (slightly) more optimal approach has the disadvantage that you don't
know with certainty when you have reached the
Perhaps other, more experienced and reputable contributors to this list can
comment but to be frank: Cassandra is probably not for you (at least for now).
I personally feel Cassandra is one of the stronger NoSQL options out there and
has the potential to become the defacto standard; but its not
is sending garbage to the others.
Either there's a bug in the bleeding edge code you are running (did
you try rc1?) or you do have nodes on different versions or you have a
hardware problem.
On Sat, Dec 4, 2010 at 5:51 PM, Dan Hendry dan.hendry.j...@gmail.com
wrote:
Here are two other errors
One of my Cassandra nodes is giving me a number of errors then effectively
dying. I think it was somehow caused by interrupting a nodetool clean
operation. Running a recent 0.7 build out of svn.
ERROR [MutationStage:26] 2010-12-04 16:23:04,395 RowMutationVerbHandler.java
(line 83) Error in row
to be problems reading from the node. At the very
least, read performance is massively degraded.
On Sat, Dec 4, 2010 at 5:52 PM, Dan Hendry dan.hendry.j...@gmail.comwrote:
One of my Cassandra nodes is giving me a number of errors then effectively
dying. I think it was somehow caused
)
On Sat, Dec 4, 2010 at 6:29 PM, Dan Hendry dan.hendry.j...@gmail.comwrote:
No, all nodes are running very recent ( 2 day old) code out of the 0.7
branch. This cluster has always had 0.7 RC1(+) code running on it
On Sat, Dec 4, 2010 at 6:24 PM, Jonathan Ellis jbel...@gmail.com wrote:
Are you
I am seeing fairly strange, behavior in my Cassandra cluster.
Setup
- 3 nodes (lets call them nodes 1 2 and 3)
- RF=2
- A set of servers (producers) which which write data to the cluster at
consistency level ONE
- A set of servers (consumers/processors) which read data from the cluster
at
guaranteed to see it on the first.
This was fixed in 0.6.4 but apparently I botched the merge to the 0.7
branch. I corrected that just now, so when you update, you should be
good to go.
On Fri, Dec 3, 2010 at 9:19 PM, Dan Hendry dan.hendry.j...@gmail.com
wrote:
I am seeing fairly strange
what is going on? Could the CPU usage STILL be related to the
repair? Is there any way to check? I hesitate to simply kill the node given
the 14 outstanding log message and as doing so has caused me problems in
the past when using beta versions.
Dan Hendry
unbelievable, is the fact Cassandra
cant seem to recover from the error and I am loosing data.
Dan Hendry
be appreciated. Thanks,
Dan Hendry
:
@param ttl. An optional, positive delay (in seconds) after which the column
will be automatically deleted.
Augi
2010/10/6 Dan Hendry d...@ec2.dustbunnytycoon.com
Hi,
I have a quick and quite frankly ridiculous question regarding the column
TTL value; what are the time units
68 matches
Mail list logo
