SurgeCon 2012

2012-09-05 Thread Chris Burroughs
Surge [1] is scalability focused conference in late September hosted in Baltimore. It's a pretty cool conference with a good mix of operationally minded people interested in scalability, distributed systems, systems level performance and good stuff like that. You should go! [2] For those of you

Cassandra Meetup in DC

2011-05-02 Thread Chris Burroughs
http://www.meetup.com/Cassandra-DC-Meetup/ *What*: First Cassandra DC Meetup *When*: Thursday, May 12, 2011 at 6:30 PM *Where*: Northside Social Coffee Wine - 3211 Wilson Blvd Arlington, VA I'm pleased to announce the the first Cassandra DC Meetup

Re: Native heap leaks?

2011-05-05 Thread Chris Burroughs
On 2011-05-05 06:30, Hannes Schmidt wrote: This was my first thought, too. We switched to mmap_index_only and didn't see any change in behavior. Looking at the smaps file attached to my original post, one can see that the mmapped index files take up only a minuscule part of RSS. I have not

Re: Index interval tuning

2011-05-10 Thread Chris Burroughs
On 05/10/2011 02:12 PM, Peter Schuller wrote: That reminds me, my false positive ration is stuck at 1.0, so I guess bloom filters aren't doing a lot for me. That sounds unlikely unless you're hitting some edge case like reading a particular row that happened to be a collision, and only that

Re: Index interval tuning

2011-05-11 Thread Chris Burroughs
On 05/10/2011 10:24 PM, aaron morton wrote: What version and what were the values for RecentBloomFilterFalsePositives and BloomFilterFalsePositives ? The bloom filter metrics are updated in SSTableReader.getPosition() the only slightly odd thing I can see is that we do not count a key

Re: Data directories

2011-06-09 Thread Chris Burroughs
On 06/08/2011 05:54 AM, Héctor Izquierdo Seliva wrote: Is there a way to control what sstables go to what data directory? I have a fast but space limited ssd, and a way slower raid, and i'd like to put latency sensitive data into the ssd and leave the other data in the raid. Is this possible?

Cassandra DC Upcoming Meetup

2011-06-15 Thread Chris Burroughs
Cassandra DC's first meetup of the pizza and talks variety will be on July 6th. There will be an introductory sort of presentation and a totally cool one on Pig integration. If you are in the DC area it would be great to see you there. http://www.meetup.com/Cassandra-DC-Meetup/events/22145481/

Re: BloomFilterFalsePositives equals 1.0

2011-06-22 Thread Chris Burroughs
To be precise, you made n requests for non-existent keys, got n negative responses, and BloomFilterFalsePositives also went up by n? On 06/21/2011 11:06 PM, Preston Chang wrote: Hi,all: I have a problem with bloom filter. When made a test which tried to get some nonexistent keys, it

Re: OOM (or, what settings to use on AWS large?)

2011-06-22 Thread Chris Burroughs
On 06/22/2011 08:53 AM, Sasha Dolgy wrote: Yes ... this is because it was the OS that killed the process, and wasn't related to Cassandra crashing. Reviewing our monitoring, we saw that memory utilization was pegged at 100% for days and days before it was finally killed because 'apt' was

Re: 99.999% uptime - Operations Best Practices?

2011-06-22 Thread Chris Burroughs
On 06/22/2011 05:33 PM, Les Hazlewood wrote: Just to be clear: I understand that resources like [1] and [2] exist, and I've read them. I'm just wondering if there are any 'gotchas' that might be missing from that documentation that should be considered and if there are any recommendations

Re: OOM (or, what settings to use on AWS large?)

2011-06-22 Thread Chris Burroughs
...and the buffered/cached seems much better -sd On Thu, Jun 23, 2011 at 12:09 AM, Chris Burroughs chris.burrou...@gmail.com wrote: In `free` terms, by pegged do you mean that free Mem was 0, or -/+ buffers/cache as 0?

Re: 99.999% uptime - Operations Best Practices?

2011-06-23 Thread Chris Burroughs
On 06/22/2011 10:03 PM, Edward Capriolo wrote: I have not read the original thread concerning the problem you mentioned. One way to avoid OOM is large amounts of RAM :) On a more serious note most OOM's are caused by setting caches or memtables too large. If the OOM was caused by a software

Re: 99.999% uptime - Operations Best Practices?

2011-06-23 Thread Chris Burroughs
On 06/22/2011 07:12 PM, Les Hazlewood wrote: Telling me to read the mailing lists and follow the issue tracker and use monitoring software is all great and fine - and I do all of these things today already - but this is a philosophical recommendation that does not actually address my question.

Re: 99.999% uptime - Operations Best Practices?

2011-06-23 Thread Chris Burroughs
On 06/23/2011 01:56 PM, Les Hazlewood wrote: Is there a roadmap or time to 1.0? Even a ballpark time (e.g next year 3rd quarter, end of year, etc) would be great as it would help me understand where it may lie in relation to my production rollout. The C* devs are rather strongly inclined

Re: Cassandra DC Upcoming Meetup

2011-07-05 Thread Chris Burroughs
On 06/15/2011 08:57 AM, Chris Burroughs wrote: Cassandra DC's first meetup of the pizza and talks variety will be on July 6th. There will be an introductory sort of presentation and a totally cool one on Pig integration. If you are in the DC area it would be great to see you there. http

Re: Storing counters in the standard column families along with non-counter columns ?

2011-07-11 Thread Chris Burroughs
On 07/10/2011 01:09 PM, Aditya Narayan wrote: Is there any target version in near future for which this has been promised ? The ticket is problematic in that it would -- unless someone has a clever new idea -- require breaking thrift compatibility to add it to the api. Since is unfortunate

Survey: Cassandra/JVM Resident Set Size increase

2011-07-12 Thread Chris Burroughs
### Preamble There have been several reports on the mailing list of the JVM running Cassandra using too much memory. That is, the resident set size is (max java heap size + mmaped segments) and continues to grow until the process swaps, kernel oom killer comes along, or performance just degrades

Re: Storing counters in the standard column families along with non-counter columns ?

2011-07-14 Thread Chris Burroughs
On 07/13/2011 03:57 PM, Aaron Morton wrote: You can always use a dedicated CF for the counters, and use the same row key. Of course one could do this. The problem is you are now spending ~2x disk space on row keys, and app specific client code just became more complicated.

Re: JNA to avoid swap but physical memory increase

2011-07-15 Thread Chris Burroughs
On 07/15/2011 07:24 AM, Daniel Doubleday wrote: Also our experience shows that the jna call does not prevent swapping so the general advice is disable swap. Can you confirm you don't get the (paraphrasing) whoops we tried mlockall but ulimits denied us message on startup?

Re: cassandra server disk full

2011-07-29 Thread Chris Burroughs
On 07/25/2011 01:53 PM, Ryan King wrote: Actually I was wrong– our patch will disable gosisp and thrift but leave the process running: https://issues.apache.org/jira/browse/CASSANDRA-2118 If people are interested in that I can make sure its up to date with our latest version. Thanks

Re: Survey: Cassandra/JVM Resident Set Size increase

2011-07-29 Thread Chris Burroughs
?bug_id=7066129) is the culprit. On 07/12/2011 09:28 AM, Chris Burroughs wrote: ### Preamble There have been several reports on the mailing list of the JVM running Cassandra using too much memory. That is, the resident set size is (max java heap size + mmaped segments) and continues to grow

Surgecon Meetup?

2011-09-25 Thread Chris Burroughs
Surge [1] is scalability focused conference in late September hosted in Baltimore. It's a pretty cool conference with a good mix of operationally minded people interested in scalability, distributed systems, systems level performance and good stuff like that. You should go! [2] Anyway, I'll be

Re: Surgecon Meetup?

2011-09-27 Thread Chris Burroughs
/2011 07:27 PM, Chris Burroughs wrote: Surge [1] is scalability focused conference in late September hosted in Baltimore. It's a pretty cool conference with a good mix of operationally minded people interested in scalability, distributed systems, systems level performance and good stuff like

ApacheCon meetup?

2011-10-04 Thread Chris Burroughs
ApacheCon NA is coming up next month. I suspect there will be at least a few Cassandra users there (yeah new release!). Would anyone be interested in getting together and sharing some stories? This could either be a official [1] meetup. Or grabbing food together sometime. [1]

Re: ApacheCon meetup?

2011-10-12 Thread Chris Burroughs
On 10/11/2011 12:05 PM, Eric Evans wrote: Let's do it. We can organize an official one, and still grab food together if that's not enough. :) Great! Thanks for putting this together.

Re: CMS GC initial-mark taking 6 seconds , bad?

2011-10-20 Thread Chris Burroughs
On 10/20/2011 09:38 AM, Maxim Potekhin wrote: I happen to have 48GB on each machines I use in the cluster. Can I assume that I can't really use all of this memory productively? Do you have any suggestion related to that? Can I run more than one instance on Cassandra on the same box (using

Re: Second Cassandra users survey

2011-11-14 Thread Chris Burroughs
- It would be super cool if all of that counter work made it possible to support other atomic data types (sets? CAS? just pass a assoc/commun Function to apply). - Again with types, pluggable type specific compression. - Wishy washy wish: Simpler elasticity I would like to go from 6--8--7

Re: 1.0.3 CLI oddities

2011-12-11 Thread Chris Burroughs
Sounds like https://issues.apache.org/jira/browse/CASSANDRA-3558 and the other tickets reference there. On 11/28/2011 05:05 AM, Janne Jalkanen wrote: Hi! (Asked this on IRC too, but didn't get anyone to respond, so here goes...) Is it just me, or are these real bugs? On 1.0.3, from

Re: Row caching in Cassandra 1.1 by column family

2012-06-18 Thread Chris Burroughs
Check out the rows_cached CF attribute. On 06/18/2012 06:01 PM, Oleg Dulin wrote: Dear distinguished colleagues: I don't want all of my CFs cached, but one in particular I do. How can I configure that ? Thanks, Oleg

Re: Distinct Counter Proposal for Cassandra

2012-06-29 Thread Chris Burroughs
On 06/13/2012 01:00 PM, Yuki Morishita wrote: The above implementation and most of the other ones (including stream-lib) implement the optimized version of the algorithm which counts up to 10^9, so may need some work. Other alternative is self-learning bitmap

Re: Distinct Counter Proposal for Cassandra

2012-06-29 Thread Chris Burroughs
Well I obviously think it would be handy. If this get's proposed and end's up using stream-lib don't be shy about asking for help. On a more general note, it would be great to see the special case Counter code become more general atomic operation code. On 06/13/2012 01:15 PM, Utku Can Topçu

Re: Experiences with Cassandra hardware planning

2010-10-25 Thread Chris Burroughs
On 10/24/2010 11:16 PM, Eric Rosenberry wrote: I wanted to share back to the community some of the learnings we have come across including the hardware configuration we have been successful with (YMMV). This is still a work in progress naturally. I have written up a detailed blog post about

Confused about CASSANDRA-1417; saving row cache

2011-01-11 Thread Chris Burroughs
https://issues.apache.org/jira/browse/CASSANDRA-1417 http://www.riptano.com/blog/whats-new-cassandra-066 My naive reading of CASSANDRA-1417 was that it could be used to save the row cache to disk. Empirically it appears to only save the row keys, and then reads each row. In my case I set the

Re: Confused about CASSANDRA-1417; saving row cache

2011-01-11 Thread Chris Burroughs
On 01/11/2011 12:23 PM, Peter Schuller wrote: Is this the intentional implementation? Are there any reason not to just the entire row to disk to allow for faster startup? Intentional (in the sense of not a mistake), but see: https://issues.apache.org/jira/browse/CASSANDRA-1625 The

Re: Confused about CASSANDRA-1417; saving row cache

2011-01-11 Thread Chris Burroughs
On 01/11/2011 02:56 PM, Peter Schuller wrote: But now I need two knobs: Max size of row cache (best optimal steady state hit rate) and number of row cache items to read in on startup (so that the ROW-READ-STAGE does not need to drop packets and node can be restarted in a reasonable amount of

Re: Confused about CASSANDRA-1417; saving row cache

2011-01-11 Thread Chris Burroughs
On 2011-01-11 15:41, Chris Burroughs wrote: On 01/11/2011 02:56 PM, Peter Schuller wrote: But now I need two knobs: Max size of row cache (best optimal steady state hit rate) and number of row cache items to read in on startup (so that the ROW-READ-STAGE does not need to drop packets and node

Re: Bloom filter

2011-01-13 Thread Chris Burroughs
On 01/13/2011 04:07 PM, Carlos Sanchez wrote: Could someone tell me where (what classes) or what library is Cassandra using for its bloom filters? src/java/org/apache/cassandra/utils/BloomFilter.java

reduced cached mem; resident set size growth

2011-01-27 Thread Chris Burroughs
We have a 6 node Cassandra 0.6.8 cluster running on boxes with 4 GB of RAM. Over the course of several weeks cached memory slowly decreases until Cassandra is restarted or something bad happens (ie oom killer). Performance obviously suffers as cached memory is no longer available. Here is a graph

Re: reduced cached mem; resident set size growth

2011-01-28 Thread Chris Burroughs
On 01/28/2011 10:51 AM, sridhar basam wrote: On Thu, Jan 27, 2011 at 12:23 PM, Chris Burroughs chris.burrou...@gmail.com wrote: java -version java version 1.6.0_20 Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode) cmd line

Re: reduced cached mem; resident set size growth

2011-01-28 Thread Chris Burroughs
On 01/28/2011 11:29 AM, Jake Luciani wrote: Are you using a row cache? if so what is it set too? in general it should not be a percentage. ColumnFamily CompareWith=UTF8Type Name=mycolumnfamily KeysCached=0 KeyCacheSavePeriodInSeconds=0 RowsCached=40

Re: reduced cached mem; resident set size growth

2011-01-28 Thread Chris Burroughs
On 01/28/2011 12:42 PM, sridhar basam wrote: What about your permgen usage? Do you track that? Use something like jstat -gc -t pid 5s 100 to track it. Or turn up verbose GC on your command line options to what is happening. http://img59.imageshack.us/img59/1056/permgen.png This is over

Re: reduced cached mem; resident set size growth

2011-01-28 Thread Chris Burroughs
On 01/28/2011 04:12 AM, Zhu Han wrote: Chris, Somebody else and I have the same problem as you, and reported it here: http://www.apacheserver.net/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-at1082970.htm [NB: It is not solved although the titles said so. Some response from

Re: reduced cached mem; resident set size growth

2011-02-02 Thread Chris Burroughs
On 01/28/2011 09:19 PM, Chris Burroughs wrote: Thanks Oleg and Zhu. I swear that wasn't a new hotspot version when I checked, but that's obviously not the case. I'll update one node to the latest as soon as I can and report back. RSS over 48 hours with java 6 update 23: http://img716

Re: reduced cached mem; resident set size growth

2011-02-02 Thread Chris Burroughs
On 02/02/2011 12:49 PM, Ryan King wrote: We're seeing a similar problem with one of our clusters (but over a longer time scale). Its possible that its not a leak, but just fragmentation. Unless you've told it otherwise, the jvm uses glibc's malloc implementation for off-heap allocations. We're

0.7.0 mx4j, get attribute

2011-02-02 Thread Chris Burroughs
I'm using 0.7.0 and experimenting with the new mx4j support. http://host:port/mbean?objectname=org.apache.cassandra.request%3Atype%3DReadStage Returns a nice pretty html page. For purposes of monitoring I would like to get a single attribute as xml. The docs [1] decribe a getattribute

Re: 0.7.0 mx4j, get attribute

2011-02-03 Thread Chris Burroughs
On 02/02/2011 01:41 PM, Ryan King wrote: On Wed, Feb 2, 2011 at 10:40 AM, Chris Burroughs chris.burrou...@gmail.com wrote: I'm using 0.7.0 and experimenting with the new mx4j support. http://host:port/mbean?objectname=org.apache.cassandra.request%3Atype%3DReadStage Returns a nice pretty

Re: 0.7.0 mx4j, get attribute

2011-02-03 Thread Chris Burroughs
On 02/03/2011 11:29 AM, Ran Tavory wrote: Try adding this to the end of the URL: ?template=identity That works, thanks!

Re: CF Read and Write Latency Histograms

2011-02-07 Thread Chris Burroughs
On 02/04/2011 12:43 PM, Jonathan Ellis wrote: Can you create a ticket? I noticed the same thing. CASSANDRA-2123 created.

Re: OOM during batch_mutate

2011-02-08 Thread Chris Burroughs
On 02/07/2011 06:05 PM, Jonathan Ellis wrote: Sounds like the keyspace was created on the 32GB machine, so it guessed memtable sizes that are too large when run on the 16GB one. Use update column family from the cli to cut the throughput and operations thresholds in half, or to 1/4 to be

Re: Out of control memory consumption

2011-02-09 Thread Chris Burroughs
On 02/09/2011 11:15 AM, Huy Le wrote: There is already an email thread on memory issue on this email list, but I creating a new thread as we are experiencing a different memory consumption issue. We are 12-server cluster. We use random partitioner with manually generated server tokens.

Re: Default Listen Port

2011-02-09 Thread Chris Burroughs
On 02/09/2011 04:00 PM, jeremy.truel...@barclayscapital.com wrote: What's the easiest way to change the port nodes listen for comm on from other nodes? It appears that the default is 8080 which collides with my tomcat server on one of our dev boxes. I tried doing something in cassandra.yaml

Re: Column name size

2011-02-11 Thread Chris Burroughs
On 02/11/2011 05:06 AM, Patrik Modesto wrote: Hi all! I'm thinking if size of a column name could matter for a large dataset in Cassandra (I mean lots of rows). For example what if I have a row with 10 columns each has 10 bytes value and 10 bytes name. Do I have half the row size just of

Re: Reducing memory footprint

2011-03-04 Thread Chris Burroughs
On 03/04/2011 01:53 PM, Casey Deccio wrote: I have a small ring of cassandra nodes that have somewhat limited memory capacity for the moment. Cassandra is eating up all the memory on these nodes. I'm not sure where to look first in terms of reducing the foot print. Keys cached? Compaction?

Re: OOM exceptions

2011-03-04 Thread Chris Burroughs
- Does this occur only during compaction or at seemingly random times? - How large is your heap? What jvm settings are you using? How much physical RAM do you have? - Do you have the row and/or key cache enabled? How are they configured? How large are they when the OOM is thrown? On 03/04/2011

Re: OOM exceptions

2011-03-04 Thread Chris Burroughs
See also: http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors On 03/04/2011 03:05 PM, Chris Burroughs wrote: - Does this occur only during compaction or at seemingly random times? - How large is your heap? What jvm settings are you using? How much physical

Re: OOM exceptions

2011-03-04 Thread Chris Burroughs
? On 03/04/2011 03:25 PM, Mark wrote: This happens during compaction and we are not using the RowsCached attribute. Our initial/max heap are 2 and 6 respectively and we have 8 gigs in these machines. Thanks On 3/4/11 12:05 PM, Chris Burroughs wrote: - Does this occur only during

Re: cassandra in-production experiences with .7 series

2011-03-07 Thread Chris Burroughs
On 03/05/2011 05:27 PM, Paul Pak wrote: Hello all, I was wondering if people could share their overall experiences with using .7 series of Cassandra in production? Is anyone using it? For what it's worth we are using a dozen node 0.7.x cluster have not had any major problems (our uses

Re: Reducing memory footprint

2011-03-07 Thread Chris Burroughs
On 03/04/2011 03:51 PM, Casey Deccio wrote: Are you saying: that you want a smaller heap and what settings to change to accommodate that, or that you have already set a small heap of x and Cassandra is using significantly more than that? Based on my observation above, the latter. Casey

Re: On 0.6.6 to 0.7.3 migration, DC-aware traffic and minimising data transfer

2011-03-14 Thread Chris Burroughs
On 03/11/2011 03:46 PM, Jonathan Ellis wrote: Repairs is not yet WAN-optimized but is still cheap if your replicas are close to consistent since only merkle trees + inconsistent ranges are sent over the network. What is the ticket number for WAN optimized repair?

Re: How to determine if repair need to be run

2011-03-30 Thread Chris Burroughs
On 03/29/2011 01:18 PM, Peter Schuller wrote: (What *would* be useful perhaps is to be able to ask a node for the time of its most recently started repair, to facilitate easier comparison with GCGraceSeconds for monitoring purposes.) I concur. JIRA time? (Perhaps keeping track of the same

Re: IndexInterval Tuning

2011-04-05 Thread Chris Burroughs
On 04/05/2011 09:57 AM, Jonathan Ellis wrote: On Tue, Apr 5, 2011 at 8:54 AM, Jonathan Ellis jbel...@gmail.com wrote: Adjusting indexinterval is unlikely to be useful on very narrow rows. (Its purpose is to make random access to _large_ rows doable.) Whoops, that's column_index_size_in_kb.

Minor Follow-up: reduced cached mem; resident set size growth

2011-04-05 Thread Chris Burroughs
This is a minor followup to this thread which includes required context: http://www.mail-archive.com/user@cassandra.apache.org/msg09279.html I haven't solved the problem, but since negative results can also be useful I thought I would share them. Things I tried unsuccessfully (on individual

Re: Minor Follow-up: reduced cached mem; resident set size growth

2011-04-06 Thread Chris Burroughs
On 04/05/2011 04:38 PM, Peter Schuller wrote: - Different collectors: -XX:+UseParallelGC -XX:+UseParallelOldGC Unless you also removed the -XX:+UseConcMarkSweepGC I *think* it takes precedence, so that the above options would have no effect. I didn't test. In either case, did you definitely

CL.ONE reads / RR / badness_threshold interaction

2011-04-06 Thread Chris Burroughs
My understanding for For CL.ONE. For the node that receives the request: (A) If RR is enabled and this node contains the needed row -- return immediately and do RR to remaining replicas in background. (B) If RR is off and this node contains the needed row -- return the needed data immediately.

Re: CL.ONE reads / RR / badness_threshold interaction

2011-04-07 Thread Chris Burroughs
Peter, thank you for the extremely detailed reply. To now answer my own question, the critical points that are different from what I said earlier are: that CL.ONE does prefer *one* node (which one depending on snitch) and that RR uses digests (which are not mentioned on the wiki page [1]) instead

Re: Minor Follow-up: reduced cached mem; resident set size growth

2011-04-08 Thread Chris Burroughs
On 04/05/2011 03:04 PM, Chris Burroughs wrote: I have gc logs if anyone is interested. This is from a node with standard io, jna enabled, but limits were not set for mlockall to succeed. One can see -/+ buffers/cache free shrinking and the C* pid's RSS growing. Includes several days of: gc

Analysing hotspot gc logs

2011-04-11 Thread Chris Burroughs
To avoid taking my own thread [1] off on a tangent. Does anyone have a reccomendation for a tool to graphical analysis (ie make useful graphs) out of hoptspot gc logs? Google searches have turned up several results along the lines of go try this zip file [2]. [1]

Re: CL.ONE reads / RR / badness_threshold interaction

2011-04-12 Thread Chris Burroughs
On 04/12/2011 06:27 PM, Peter Schuller wrote: So to increase pinny-ness I'll further reduce RR chance and set a badness threshold. Thanks all. Just be aware that, assuming I am not missing something, while this will indeed give you better cache locality under normal circumstances - once

Re: quick repair tool question

2011-04-12 Thread Chris Burroughs
On 04/12/2011 11:11 AM, Jonathan Colby wrote: I'm not sure if this is the kosher way to rebuild the sstable data, but it seemed to work. http://wiki.apache.org/cassandra/Operations#Handling_failure Option #3.

flashcache experimentation

2011-04-18 Thread Chris Burroughs
https://github.com/facebook/flashcache/ FlashCache is a general purpose writeback block cache for Linux. We have a case where: - Access to data is not uniformly random (let's say Zipfian). - The hot set RAM. - Size of disk is such that buying enough SSDs, fast drives, multiple drives, etc

multi-dc clusters with 'local' ips and no vpn

2013-06-17 Thread Chris Burroughs
Cassandra makes the totally reasonable assumption that the entire cluster is in one routable address space. We unfortunately had a situation where: * nodes can talk to each other in the same dc on an internal address, but not talk to each other over their external 1:1 NAT address. * nodes can

Re: row cache

2013-09-03 Thread Chris Burroughs
On 09/01/2013 03:06 PM, Faraaz Sareshwala wrote: Yes, that is correct. The SerializingCacheProvider stores row cache contents off heap. I believe you need JNA enabled for this though. Someone please correct me if I am wrong here. The ConcurrentLinkedHashCacheProvider stores row cache contents

Multi-dc restart impact

2013-09-05 Thread Chris Burroughs
We have a 2 DC cluster running cassandra 1.2.9. They are in actual physically separate DCs on opposite coasts of the US, not just logical ones. The primary use of this cluster is CL.ONE reads out of a single column family. My expectation was that in such a scenario restarts would have

Re: Multi-dc restart impact

2013-09-18 Thread Chris Burroughs
On 09/17/2013 04:44 PM, Robert Coli wrote: On Thu, Sep 5, 2013 at 6:14 AM, Chris Burroughs chris.burrou...@gmail.comwrote: We have a 2 DC cluster running cassandra 1.2.9. They are in actual physically separate DCs on opposite coasts of the US, not just logical ones. The primary use

Re: I don't understand shuffle progress

2013-09-18 Thread Chris Burroughs
On 09/17/2013 09:41 PM, Paulo Motta wrote: So you're saying the only feasible way of enabling VNodes on an upgraded C* 1.2 is by doing fork writes to a brand new cluster + bulk load of sstables from the old cluster? Or is it possible to succeed on shuffling, even if that means waiting some weeks

Re: I don't understand shuffle progress

2013-09-18 Thread Chris Burroughs
it finishes I can remove the original. Any good resource that explains how to add a new datacenter to a live single dc cluster that anybody can recommend? On Wed, Sep 18, 2013 at 9:58 AM, Chris Burroughs chris.burrou...@gmail.comwrote: On 09/17/2013 09:41 PM, Paulo Motta wrote: So you're saying

Re: Nodes separating from the ring

2013-09-23 Thread Chris Burroughs
I have observed one problem with an inconsistent ring that is superficially similar (node thinks it's up but peers disagree) and noted details in CASSANDRA-6082. However, it does not sound like the details of either the symptoms, or the resolution match what you describe. If you have not

gossip settling and bootstrap problems

2013-10-07 Thread Chris Burroughs
I've been running into a variety of tricky to diagnose problems recently that could be summarized as bootstrap related tasks fail without extra hacky sleep time. This is a sample edited log file for bootstrapping a node that captures the general dynamics: http://pastebin.com/yeN9USLt This

Re: Multi-dc restart impact

2013-10-09 Thread Chris Burroughs
* perf in other DC: http://i.imgur.com/gEMrLyF.png?1 Not only was a single slow node able to cause an order of magnitude performance hit in a dc, but the other dc faired *worse*. On 09/18/2013 08:50 AM, Chris Burroughs wrote: On 09/17/2013 04:44 PM, Robert Coli wrote: On Thu, Sep 5, 2013 at 6

Re: Multi-dc restart impact

2013-10-10 Thread Chris Burroughs
Thanks, double checked; reads are CL.ONE. On 10/10/2013 11:15 AM, J. Ryan Earl wrote: Are you doing QUORUM reads instead of LOCAL_QUORUM reads? On Wed, Oct 9, 2013 at 7:41 PM, Chris Burroughs chris.burrou...@gmail.comwrote: I have not been able to do the test with the 2nd cluster, but have

vnode + multi dc migration

2013-10-11 Thread Chris Burroughs
I know there is a good deal of interest [1] on feasible methods for enabling vnodes on clusters that did not start with them. We recently completed a migration from a production cluster not using vnodes and in a single DC to one using vnodes in two DCs. We used the just spin up a new DC and

Re: The performance difference of online bulk insertion and the file-based bulk loading

2013-10-23 Thread Chris Burroughs
On 10/15/2013 08:41 AM, José Elias Queiroga da Costa Araújo wrote: - is that is there a way that we can warm-up the cache, after the file-based bulk loading, so that we can allow the data to be cached first in the memory, and then afterwards, when we issue the bulk retrieval, the performance can

Re: nodetool status reporting dead node as UN

2013-10-23 Thread Chris Burroughs
When debugging gossip related problems (is this node really down/dead/some-werid state) you might have better luck looking at `nodetool gossipinfo`. The UN even though everything is bad thing might be https://issues.apache.org/jira/browse/CASSANDRA-5913 I'm not sure what exactly what

Re: Huge multi-data center latencies

2013-10-23 Thread Chris Burroughs
On 10/21/2013 07:03 PM, Hobin Yoon wrote: Another question is how do you get the local DC name? Have a look at org.apache.cassandra.db.EndpointSnitchInfo.getDatacenter

Re: How to use Cassandra on-node storage engine only?

2013-10-23 Thread Chris Burroughs
As far as I know this had not been done before. I would be interested in hearing how it turned out. On 10/23/2013 09:47 AM, Yasin Celik wrote: I am developing an application for data storage. All the replication, routing and data retrieving types of business are handled in my application.

Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled

2013-11-06 Thread Chris Burroughs
Both caches involve several objects per entry (What do we want? Packed objects. When do we want them? Now!). The size is an estimate of the off heap values only and not the total size nor number of entries. An acceptable size will depend on your data and access patterns. In one case we

Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled

2013-11-07 Thread Chris Burroughs
On 11/06/2013 11:18 PM, Aaron Morton wrote: The default row cache is of the JVM heap, have you changed to the ConcurrentLinkedHashCacheProvider ? ConcurrentLinkedHashCacheProvider was removed in 2.0.x.

Re: Endless loop LCS compaction

2013-11-08 Thread Chris Burroughs
On 11/07/2013 06:48 AM, Desimpel, Ignace wrote: Total data size is only 3.5GB. Column family was created with SSTableSize : 10 MB You may want to try a significantly larger size. https://issues.apache.org/jira/browse/CASSANDRA-5727

Re: Why truncate previous hints when upgrade from 1.1.9 to 1.2.6?

2013-11-08 Thread Chris Burroughs
NEWS.txt has some details and suggested procedures - The hints schema was changed from 1.1 to 1.2. Cassandra automatically snapshots and then truncates the hints column family as part of starting up 1.2 for the first time. Additionally, upgraded nodes will not store new

Re: Cassandra 1.1.6 - New node bootstrap not completing

2013-11-08 Thread Chris Burroughs
On 11/01/2013 03:03 PM, Robert Coli wrote: On Fri, Nov 1, 2013 at 9:36 AM, Narendra Sharma narendra.sha...@gmail.comwrote: I was successfully able to bootstrap the node. The issue was RF 2. Thanks again Robert. For the record, I'm not entirely clear why bootstrapping two nodes into the

MiscStage Backup

2013-11-26 Thread Chris Burroughs
I'm trying to debug a node that has a backup in MiscStage. Starting a bit under 24 hours ago the number of Pending tasks jumped to a bit under 400 and hovered around there. It looks like repair requests from other nodes (tpstats on this node shows AntiEntropySessions: 0, 0, 0, which I think

Re: How to measure data transfer between data centers?

2013-12-04 Thread Chris Burroughs
https://wiki.apache.org/cassandra/Metrics has per node Streaming metrics that include total bytes/in out. That is only a small bit of what you want though. For total DC bandwidth it might be more straightforward to measure this at the router/switch/fancy-network-gear level. On 12/03/2013

Re: vnode in production

2014-01-06 Thread Chris Burroughs
On 01/02/2014 01:51 PM, Arindam Barua wrote: 1. the stability of vnodes in production I'm happily using vnodes in production now, but I would have trouble calling them stable for more than small clusters until very recently (1.2.13). CASSANDRA-6127 served as a master ticket for most of

Re: vnode in production

2014-01-06 Thread Chris Burroughs
On 01/06/2014 01:56 PM, Arindam Barua wrote: Thanks for your responses. We are on 1.2.12 currently. The fixes in 1.2.13 seem to help for clusters in the 500+ node range (like CASSANDRA-6409). Ours is below 50 now, so we plan to go ahead and enable vnodes with the 'add a new DC' procedure. We

nodetool cleanup / TTL

2014-01-07 Thread Chris Burroughs
This has not reached a consensus in #cassandra in the past. Does `nodetool cleanup` also remove data that has expired from a TTL?

Re: nodetool cleanup / TTL

2014-01-07 Thread Chris Burroughs
On 01/07/2014 01:38 PM, Tyler Hobbs wrote: On Tue, Jan 7, 2014 at 7:49 AM, Chris Burroughs chris.burrou...@gmail.comwrote: This has not reached a consensus in #cassandra in the past. Does `nodetool cleanup` also remove data that has expired from a TTL? No, cleanup only removes rows

Re: Row cache vs. OS buffer cache

2014-01-23 Thread Chris Burroughs
My experience has been that the row cache is much more effective. However, reasonable row cache sizes are so small relative to RAM that I don't see it as a significant trade-off unless it's in a very memory constrained environment. If you want to enable the row cache (a big if) you probably

Re: Question about local reads with multiple data centers

2014-02-06 Thread Chris Burroughs
On 01/29/2014 08:07 PM, Donald Smith wrote: My question: will the read process try to read first locally from the datacenter DC2 I specified in its connection string? I presume so. (I doubt that it uses the client's IP address to decide which datacenter is closer. And I am unaware of

Re: what tool will create noncql columnfamilies in cassandra 3a

2014-02-06 Thread Chris Burroughs
On 02/05/2014 04:57 AM, Sylvain Lebresne wrote: How will users adjust the meta data of non cql column families The rational for removing cassandra-cli is mainly that maintaining 2 fully featured command line interface is a waste of the project resources in the long run. It's just a tool using

Re: First SSTable file is not being compacted

2014-02-06 Thread Chris Burroughs
On 02/06/2014 01:17 AM, Sameer Farooqui wrote: I'm running C* 2.0.4 and when I have a handful of SSTable files and trigger a manual compaction with 'nodetool compact' the first SSTable file doesn't get compacted away. Is there something special about the first SSTable that it remains even after

  1   2   >