to disablegossip and make other nodes not send
requests to it. disabling thrift would also be advised, or even
firewalling it prior to restart.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
I can recommend Jolokia highly for providing an HTTP/JSON interface to
JMX (it can be trivially run in agent mode by just altering JVM args):
http://www.jolokia.org/
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
amounts of data? Large or many columns (or
both), etc. Essentially all working data that your request touches
is allocated on the heap and contributes to allocation rate and ParNew
frequency.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
is
not as compact as PostgreSQL. For example column names are duplicated
in each row, and the row key is duplicated twice (once in index, once
in data).
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
I think that was clear from your post. I don't see a problem with your
process. Setting gc grace to 0 and forcing compaction should indeed
return you to the smallest possible on-disk size.
(But may be unsafe as documented; can cause deleted data to pop back up, etc.)
--
/ Peter Schuller
over all rows:
for row_id, row in your_column_family.get_range():
https://github.com/pycassa/pycassa
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
This problem is not new to 1.1.
On Sep 6, 2012 5:51 AM, Radim Kolar h...@filez.com wrote:
i would migrate to 1.0 because 1.1 is highly unstable.
Has anyone tried running 1.1.1 on Java 7?
Have been running jdk 1.7 on several clusters on 1.1 for a while now.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
by inter-region pointers). If you can avoid that, one might
hope to avoid full gc:s all-together.
The jury is still out on my side; but like I said, I've seen promising
indications.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
question is how often. But given the lack of handling of such failure
modes, the effect on clients is huge. Recommend data reads by default
to mitigate this and a slew of other sources of problems (and for
counter increments, we're rolling out least-active-request routing).
--
/ Peter Schuller
it is
in action.
FWIW, J9's balanced collector is very similar to G1 in it's design.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Our full gc:s are typically not very frequent. Few days or even weeks
in between, depending on cluster.
*PER NODE* that is. On a cluster of hundreds of nodes, that's pretty
often (and all it takes is a single node).
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
sstable will effectively cover almost the entire range
(since you're effectively spraying random tokens at it, unless clients
are writing data in md5 order).
(Maybe it's different for ordered partitioning though.)
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
the causes of un-predictable behavior w.r.t.
GC by being careful about it's memory allocation and *retention*
profile. For the specific case of avoiding *ever* seeing a full gc, it
gets even more complex.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
the top of my head) the resulting value being correct
is if the later increment (N2 in this case) is somehow including N1 as
well (e.g., because it was generated by first reading the current
counter value).
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
be safely retried. Cassandra counters are generally not
useful if *strict* correctness is desired, for this reason.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
with slightly changed
workloads? It's very hard to blackbox-test GC settings, which is
probably why GC tuning can be perceived as a useless game of
whack-a-mole.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
a single sstable bigger
than what would normally happen, and it takes more total disk space
before it will be part of a compaction again.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
of cassandra?
It's in the 1.1 branch; I don't remember if it went into a release
yet. If not, it'll be in the next 1.1.x release.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
On Oct 22, 2012 11:54 AM, B. Todd Burruss bto...@gmail.com wrote:
does nodetool cleanup perform a major compaction in the process of
removing unwanted data?
No.
FWIW, we're using openjdk7 on most of our clusters. For those where we
are still on openjdk6, it's not because of an issue - just haven't
gotten to rolling out the upgrade yet.
We haven't had any issues that I recall with upgrading the JDK.
--
/ Peter Schuller (@scode, http
of a different story and if you want to
test behavior when nodes go down I suggest including that. See
CASSANDRA-2540 and CASSANDRA-3927.)
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
settings
(typically trading pollution of page cache vs. number of I/O:s).
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
to skip rows
that are obviously bad, but true integrity checking is not supported
at this time.
--
/ Peter Schuller
. the common case of wanting to listen on 127.0.0.1 but no public
interfaces...
--
/ Peter Schuller
haywire as you don't
service as many I/O requests as are coming in.
There is a grey area in between where latency will be very sensitive
to smallish changes in I/O load but aggregate throughput remaining
below what can be sustained.
--
/ Peter Schuller
.
--
/ Peter Schuller
usage during periods of timeouts. If the huge allocations fail due to
fragmentation and fallback to Full GC that might be an expected
result. Else -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps.
--
/ Peter Schuller
issues. *Maybe* after two full gc:s tops if the
first happens while there's a mix still active in memtables.
--
/ Peter Schuller
Thanks Peter. I am using java version of the stress testing tool from the
contrib folder. Is there any issue that should be aware of? Do you recommend
using pystress?
I just saw Brandon file this:
https://issues.apache.org/jira/browse/CASSANDRA-2578
Maybe that's it.
--
/ Peter Schuller
the former row key gets
restored to a point in time prior to that of the latter row key may
cause the latter write to become visible even though the former write
is lost.
--
/ Peter Schuller
: Think before typing. :)
--
/ Peter Schuller
that the need for major compactions is significantly lessened
or even eliminated. However, running major compactions won't cause
tombstones *not* to be removed; it's just not required *in order* for
them to be removed.
--
/ Peter Schuller
.
--
/ Peter Schuller
to the effects of the temporary spike in data size
and cache coldness. Sounds like it makes good sense in your situation
though.
--
/ Peter Schuller
lead to a delay
of sstable removal which will vary with whatever else is happening
(the more busy the node, the more often a concurrent mark/sweep gc
phase is triggered, and the more frequently obsolete sstables are
deleted).
--
/ Peter Schuller
to judge and likely depends a lot on i/o
scheduling and other details.
--
/ Peter Schuller
store?
So the only thing I can do is test it and see how it goes. To make the
change affective, should I do anything beyond changing the value in
cassandra.yaml and restart the node? I'll try first with 256 and see
what happens.
That should be it.
--
/ Peter Schuller
What is the best way to find keys of such big rows?
One, if not necessarily the best, way is to check system.log for large
row warnings that trigger for rows large enough to be compacted
lazily. Grep for 'azy' (or lazy case-insens) and you should find it.
--
/ Peter Schuller
setting in question is the memtable_flush_after
setting. Do you have that set to something very high on one of your
column families?
You can use describe keyspace name_of_keyspace in cassandra-cli to
check current settings.
--
/ Peter Schuller
? Including overwrites. If not, I'm
not sure what's going on. Since you said it took about a day of
traffic it feels fishy.
--
/ Peter Schuller
to be, due to the LRU:ishness of caches, the less frequently accessed
data that tends to make it difficult to judge by numbers that include
all I/O.
--
/ Peter Schuller
Ok, so I think I found one major cause contributing to the increasing
Nice job tracking this down! That is useful to know, even outside of
Cassandra use cases. Frankly it's disappointing to learn what nio is
doing.
--
/ Peter Schuller
is useful since it allows consistency
semantics similar to ALL but allows you to survive nodes being down,
at the cost of a higher RF (3 at least))
--
/ Peter Schuller
/) for
that.
--
/ Peter Schuller
/browse/CASSANDRA-2554 which may be
relevant, but simple overload is also a possible reason.
--
/ Peter Schuller
. Particularly in a
situation with lots of dropped messages.
I'm getting the 2^15 from AntiEntropyService.Validator.Validator()
which passes a maxsize of 2^15 to the MerkelTree constructor.
--
/ Peter Schuller
the more I think of it ;)
--
/ Peter Schuller
Hmmm, I'm starting to like this idea more and more the more I think of it ;)
Filed: https://issues.apache.org/jira/browse/CASSANDRA-2699
--
/ Peter Schuller
and cassandra service
5). read the key list generated in step 2) with consistency level ONE
How sure are you that the system is honoring fsync() properly,
including flushing any caches on underlying drives? Or is this with
battery backed caching RAID controllers?
--
/ Peter Schuller
will legitimiately get very low
values without it indicating anything is wrong.)
--
/ Peter Schuller
kernels) barriers at the OS level - and
the list goes on.
--
/ Peter Schuller
adding a lot of overhead in terms of disk I/O
unless your data set fits comfortably in memory.
--
/ Peter Schuller
at the same time.
--
/ Peter Schuller
changes so
I'm not sure off hand what'll happen to the auto-system.gc code in
cassandra that attempts to free space.
CASSANDRA-2521 is IMO the real solution.
--
/ Peter Schuller
It would be great if Cassandra puts this on their roadmap. There is
lot of durability benefits by incorporating dc awareness into the
write consistency equation.
You may be interested in the discussion here:
https://issues.apache.org/jira/browse/CASSANDRA-2338
--
/ Peter Schuller
a memtable is flushed can the
commit log which contains the data being flushed, be removed by
Cassandra.
--
/ Peter Schuller
when it starts
up the thrift interface - check system.log.
--
/ Peter Schuller
benefits from this. The only hard requirement is the
repair schedule relative to GC grace time, and that requirement does
not change - just be mindful of the timing of the EBS snapshots and
what that means to your repair schedule.
--
/ Peter Schuller
EBS volume atomicity is good. We've had tons of experience since EBS came
out almost 4 years ago, to back all kinds of things, including large DBs.
And thanks a lot for coming forward with production experience. That
is always useful with these things.
--
/ Peter Schuller
about freeze maybe being probabilistically useful anyway.
--
/ Peter Schuller
this in detail. I should
really start working off the backlog of those blog entries...
--
/ Peter Schuller
certain
goals, including crash consistency. It still relies on the same
fundamental properties of the underlying storage device.
--
/ Peter Schuller
choosing batch commit log sync instead of
periodic if single-node durability or post-quorum-write durability is
a concern.
--
/ Peter Schuller
--
/ Peter Schuller
/networking load and is not yet rate limited like
compaction.
In addition be aware that repair can cause disk space usage to
temporarily increase if there are significant differences to be
repaired.
--
/ Peter Schuller
to be.
--
/ Peter Schuller
must be scheduled by the operator to run regularly.
The name repair is a bit unfortunate; it is not meant to imply that
it only needs to run when something is wrong.
--
/ Peter Schuller
new sstables) which is expensive for the
usual reasons with disk I/O; it's major since it covers all data.
The data read is in fact used to calculate a merkle tree for
comparison with neighbors, as claimed.
--
/ Peter Schuller
and your memory is enough to keep the hot set, and you're disk I/O is
coming form the long tail, increasing the amount of cache to 200 gig
may not necessarily give you a huge improvement in terms of
percentages.
--
/ Peter Schuller
be 10 times that of the original cache.
I did a quick Google but didn't find a good piece describing it more
properly, but hopefully the above is helpful. Some related reading
might be http://en.wikipedia.org/wiki/Long_Tail
--
/ Peter Schuller
documented in the link I sent before, unless you have specific
reasons not to and know what you're doing.
--
/ Peter Schuller
repair take?
So basically, leave significant margin.
--
/ Peter Schuller
dependent on some marker that isn't
written until commit log synch.)
--
/ Peter Schuller (@scode on twitter)
# wait for a bit until no one is sending it writes anymore
More accurately, until all other nodes have realized it's down
(nodetool ring on each respective host).
--
/ Peter Schuller (@scode on twitter)
?
--
/ Peter Schuller (@scode on twitter)
to increase RF.
If you *really* know what you're doing and why you want RF to track
total node count, I'm sure there are *some* cases where this makes
sense. But nothing you've said so far really indicates you're in such
a position.
--
/ Peter Schuller (@scode on twitter)
really have permission
to do the mlockall()?
(Not that I disagree in any way that swap should be disabled, +1 on that.)
--
/ Peter Schuller (@scode on twitter)
.
--
/ Peter Schuller (@scode on twitter)
checking out:
https://issues.apache.org/jira/browse/CASSANDRA-1283
https://issues.apache.org/jira/browse/CASSANDRA-1969
If it can be done via that the nice thing is that you don't loose consistency.
--
/ Peter Schuller (@scode on twitter)
of consistency level *never*
affects *which* nodes are responsible for a given row key, nor does it
affect which rows will eventually receive writes. It *only* affects
how many nodes must respond before the operation (read or write) is
considered successful.
Does that make it clearer?
--
/ Peter
will be
serving the requests.
--
/ Peter Schuller (@scode on twitter)
. It is not the case
that having RF be equal to the cluster size is in and of itself a
useful property.
--
/ Peter Schuller (@scode on twitter)
decrease read performance, as the average row can become more
spread out over multiple sstables. This is one potential driver for
compaction.
--
/ Peter Schuller (@scode on twitter)
Thanks! Then does it mean that before compaction if read call comes for that
key sort is done at the read time since column b, c and a are in different
ssTables.
Essentially yes; a merge-sort happens (since they are sorted locally
in each sstable).
--
/ Peter Schuller (@scode on twitter)
with respect to.)
Clocks should be synchronized, yes. But either your data model is such
that conflicting writes are okay, or you need external co-ordination.
There's not hoping for the best by keeping clocks better in synch.
--
/ Peter Schuller (@scode on twitter)
effects on caches like streaming through all the bf and
index files, restarts are certainly detrimental to the page cache.
Also you may still see some eviction (even if it doesn't *necessarily*
happen) depending (particularly if not running with numactl set to
interleave).
--
/ Peter Schuller
nodes that run into this.
--
/ Peter Schuller (@scode on twitter)
construct a benchmark where there's no difference, yet see a
significant difference in a real-world scenario when your benchmarked
I/O is intermixed with other I/O. Not to mention subtle differences in
behaviors of kernels, RAID controllers, disk drive controllers, etc...
--
/ Peter Schuller (@scode
that cost is not specific to
mmap():ed files (once a given page is in core that is).
(But again, I'm not arguing the point in Cassandra's case; just generally.)
--
/ Peter Schuller (@scode on twitter)
-ordination outside of Cassandra, there will be the
potential for multiple clients reading and writing without awareness
of each other. Whatever the behavior is, your data model must be such
that this is acceptable.
--
/ Peter Schuller (@scode on twitter)
that ReadStage is usually full (@ your
limit)?
--
/ Peter Schuller (@scode on twitter)
with RF=3,
then for whatever ranges of the ring that node was partially
responsible for, only 2 of the 3 copies will be up/available. They do
not automatically migrate somewhere else.
--
/ Peter Schuller (@scode on twitter)
The compactions ettings do not affect repair. (Thinking out loud, or does it
? Validation compactions and table builds.)
It does.
--
/ Peter Schuller (@scode on twitter)
filters and indexes. Can be CPU or I/O bound (or throttled) -
nodetool compactionstats, htop, iostat -x -k 1
--
/ Peter Schuller (@scode on twitter)
pasted; periodic
repairs are part of regular cluster maintenance. See:
http://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair
--
/ Peter Schuller (@scode on twitter)
netstats. If there are stuck streams, they might be causing sstable
retention beyond what you'd expect.
--
/ Peter Schuller (@scode on twitter)
rebasing is necessary. You might try a
trunk from further back in time (around the time Stu submitted the
patch).
I'm not quite sure what you're actual problem is though, if it's
source code access then the easiest route is probably to check it out
from https://github.com/apache/cassandra
--
/ Peter
) that
indicates the items you specifically say are processed twice, were in
fact written twice to Cassandra?
--
/ Peter Schuller (@scode on twitter)
is if there
is a very small amount of data (yet non-zero), or significant amounts.
--
/ Peter Schuller (@scode on twitter)
1 - 100 of 552 matches
Mail list logo
