Re: java.lang.StackOverflowError with big IN list

It must be a very large IN clause, which is probably not advisable. But it shouldn't cause this error, and since it's an easy fix to prevent it, if you file a JIRA I'll post a patch. On 10 January 2014 13:08, Дмитрий Шохов sho...@gmail.com wrote: Hello I'm getting stack overflow when running

Re: Weird GC

It's possible the time attributed to GC is actually spent somewhere else; a multitude of tasks may occur during the same safepoint as a GC. We've seen some batch revoke of biased locks take a long time, for instance; *if* this is happening in your case, and we can track down which objects, I would

Re: Intermittent long application pauses on nodes

Frank, The same advice for investigating holds: add the VM flags -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 (you could put something above 1 there, to reduce the amount of logging, since a pause of 52s will be pretty obvious even if aggregated with lots of other safe

Re: Intermittent long application pauses on nodes

and send all stdout to a log file and tell cassandra not to close stdout? Also, we have swap turned off as recommended. thanks On Wed, Jan 29, 2014 at 3:39 PM, Benedict Elliott Smith belliottsm...@datastax.com wrote: Frank, The same advice for investigating holds: add the VM flags -XX

Re: Weird GC

. I've added those flags as well as some others I saw in another thread that redirects stdout to a file. What information is it that you need? 2014-01-29 Benedict Elliott Smith belliottsm...@datastax.com: It's possible the time attributed to GC is actually spent somewhere else; a multitude

Re: Intermittent long application pauses on nodes

, I see a strange dependency failed error relating to JMX in the JVM stdout log file. thanks On Wed, Jan 29, 2014 at 4:44 PM, Benedict Elliott Smith belliottsm...@datastax.com wrote: Add some more flags: -XX:+UnlockDiagnosticVMOptions -XX:LogFile=${path} -XX:+LogVMOutput I never

Re: Lots of deletions results in death by GC

Is it possible you are generating *exclusively* deletes for this table? On 5 February 2014 00:10, Robert Wille rwi...@fold3.com wrote: I ran my test again, and Flush Writer's All time blocked increased to 2 and then shortly thereafter GC went into its death spiral. I doubled

Re: Lots of deletions results in death by GC

February 2014 15:09, Robert Wille rwi...@fold3.com wrote: Yes. It's kind of an unusual workload. An insertion phase followed by a deletion phase, generally not overlapping. From: Benedict Elliott Smith belliottsm...@datastax.com Reply-To: user@cassandra.apache.org Date: Tuesday, February 4

Re: Lots of deletions results in death by GC

this bug and fixed it. Now I just need to figure out where to go from here: do I wait, use the dev branch or work around. Robert From: Benedict Elliott Smith belliottsm...@datastax.com Reply-To: user@cassandra.apache.org Date: Wednesday, February 5, 2014 at 8:32 AM To: user@cassandra.apache.org

Re: impact of update operation to read operation

Also, a major compaction doesn't flush the memtable. If the memtable is still full, reads may take slightly longer as they may have to be merged with any on-disk data before being served. On 10 February 2014 21:18, Tupshin Harper tups...@tupshin.com wrote: You don't mention disks and RAM, but

Re: Intermittent long application pauses on nodes

-To: user@cassandra.apache.org user@cassandra.apache.org Date: Monday, February 3, 2014 at 6:10 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: Intermittent long application pauses on nodes On Mon, Feb 3, 2014 at 8:52 AM, Benedict Elliott Smith belliottsm...@datastax.com

Re: Intermittent long application pauses on nodes

discussed in this thread, just the G1 report wasn't very clear, sorry. regards, ondrej On Mon, Feb 17, 2014 at 11:45 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: Ondrej, It seems like your issue is much less difficult to diagnose: your collection times are long. At least

Re: Performance problem with large wide row inserts using CQL

Cassandra will throw an exception indicating the type is different than the default type. If you want untyped data, store blobs. Or store in a different column (they're free when empty, after all). Type safety is considered a good thing by many. On 20 February 2014 17:26, Peter Lin

Re: Intermittent long application pauses on nodes

cluster so I have no additional GC-logging enabled. This may be an indication that upgrading is not the (complete) solution. Regards, Joel 2014-02-17 13:41 GMT+01:00 Benedict Elliott Smith belliottsm...@datastax.com: Hi Ondrej, It's possible you were hit by the problems in this thread

Re: Intermittent long application pauses on nodes

. On 27 February 2014 17:01, Frank Ng fnt...@gmail.com wrote: We have swap disabled. Can death by paging still happen? On Thu, Feb 27, 2014 at 11:32 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: That sounds a lot like death by paging. On 27 February 2014 16:29, Frank Ng

Re: Cassandra slow on some reads

To add to this, if the 24G is the JVM limit, Cassandra will actually be using even more than this for bloom filters etc. that are managed off-heap. So the amount of page cache left is almost certainly inadequate. On 14 March 2014 13:01, Andras Szerdahelyi andras.szerdahe...@ignitionone.com

Re: Cassandra slow on some reads

Post the server logs and traces of one of the lengthy queries? On 15 Mar 2014 20:49, Batranut Bogdan batra...@yahoo.com wrote: Hello, Yes for the first query the server can be slow but no matter what is should not take 10 seconds to get one key from a column family. I can see this happening

Re: FileNotFoundException ...-Data.db

Note that drain stops gossip and thrift anyway, so they are redundant. On 25 March 2014 18:30, Robert Coli rc...@eventbrite.com wrote: On Tue, Mar 25, 2014 at 5:36 AM, Batranut Bogdan batra...@yahoo.comwrote: I am running 2.0.6 and I use /etc/init.d/cassandra start / stop . Also before

Re: any way to REALLY turn off commitlog?

Are you writing these records as batches (with begin batch, not begin unlogged)? As a logged batch ('begin batch' is logged) gets written to a system keyspace with CL before being applied to the underlying CF, and I'm not sure we want to go special casing that for non-durable writes, since you

Re: : Read a negative frame size (-2113929216)!

Vivek, The error you are seeing is a thrift error, but you say you are using the Java driver which does not operate over thrift: are you perhaps trying to connect the datastax driver to the thrift protocol port? The two protocols are not compatible, you must connect to the native_transport_port

Re: Nectar client - New Cassandra Client for .Net

The native protocol specification has always been in the Apache Cassandra repository. The implementations are not. On 2 June 2014 13:25, Peter Lin wool...@gmail.com wrote: There's nothing preventing support for native protocol going forward. It was easier to go with thrift and I happen to

Re: memtable mem usage off by 10?

If you are storing small values in your columns, the object overhead is very substantial. So what is 400Mb on disk may well be 4Gb in memtables, so if you are measuring the memtable size by the resulting sstable size, you are not getting an accurate picture. This overhead has been reduced by about

Re: memtable mem usage off by 10?

free, with ~12GB used by cassandra, the rest in caches. Reading 8k rows/s, writing 2k rows/s on a 3 node cluster. So it's not idling. BR Johan -- *From:* Benedict Elliott Smith belliottsm...@datastax.com *Sent:* Wednesday, June 4, 2014 11:56 AM

Re: memtable mem usage off by 10?

-- *From:* Benedict Elliott Smith belliottsm...@datastax.com *Sent:* Wednesday, June 4, 2014 12:33 PM *To:* user@cassandra.apache.org *Subject:* Re: memtable mem usage off by 10? These measurements tell you the amount of user data stored in the memtables, not the amount of heap used

Re: memtable mem usage off by 10?

respected. This shouldn't be possible. -- *From:* Benedict Elliott Smith belliottsm...@datastax.com *Sent:* Wednesday, June 4, 2014 1:19 PM *To:* user@cassandra.apache.org *Subject:* Re: memtable mem usage off by 10? Unfortunately it looks like the heap

Re: memtable mem usage off by 10?

of ~2GB. With a constant overhead of ~10x, as suggested, this would mean that it used 20GB, which is 2x the size of the heap. That shouldn't work. According to the OS, cassandra doesn't use more than ~11-12GB. -- *From:* Benedict Elliott Smith belliottsm

Re: memtable mem usage off by 10?

supplying it, I was assuming it was using the default. It does not exist in my config file. Sorry for the confusion. From: Benedict Elliott Smith belliottsm...@datastax.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Wednesday 4 June 2014 16:36 To: user@cassandra.apache.org

Re: memtable mem usage off by 10?

:/usr/share/cassandra/lib/thrift-server-0.3.3.jar org.apache.cassandra.service.CassandraDaemon From: Benedict Elliott Smith belliottsm...@datastax.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Wednesday 4 June 2014 17:18 To: user@cassandra.apache.org user

Re: EBS SSD - Cassandra ?

I would say this is worth benchmarking before jumping to conclusions. The network being a bottleneck (or latency causing) for EBS is, to my knowledge, supposition, and instances can be started with direct connections to EBS if this is a concern. The blog post below shows that even without SSDs the

Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?

Also https://issues.apache.org/jira/browse/CASSANDRA-7437 and https://issues.apache.org/jira/browse/CASSANDRA-7465 for rc3, although the CounterCacheKey assertion looks like an independent (though comparatively benign) bug I will file a ticket for. Can you try this against rc3 to see if the

Re: TTransportException (java.net.SocketException: Broken pipe)

Are you still seeing the same exceptions about too many open files? On Thu, Jul 17, 2014 at 6:28 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: Even after changing ulimits and moving to the recommended production settings, we are still seeing the same issue. root@lnx148-76:~# cat

Re: How to maintain the N-most-recent versions of a value?

If the versions can be guaranteed to be a adjacent (i.e. if the latest version is V, the prior version is V-1) you could issue a delete at the same time as an insert for V-N-(buffer) where buffer = 0 In general guaranteeing that is probably hard, so this seems like something that would be nice to

Re: horizontal query scaling issues follow on

How many columns are you inserting/querying per key? Could we see some example CQL statements for the insert/read workload? If you are maxing out at 10 clients, something fishy is going on. In general, though, if you find that adding nodes causes performance to degrade I would suspect that you

Re: horizontal query scaling issues follow on

like it is hurting our chances to horizontally scale with the data/schema? Thanks, Diane On Fri, Jul 18, 2014 at 6:46 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: How many columns are you inserting/querying per key? Could we see some example CQL statements for the insert

Re: Why is the cassandra documentation such poor quality?

It only takes a moment to ask to be added as a wiki contributor; if you email the dev list or ask on irc, somebody with privileges will ordinarily add you within a day. It may be a psychological barrier, but it isn't really a practical one. Still, if you feel the policy is incorrect, raise this on

Re: Why is the cassandra documentation such poor quality?

stuff on one of my blogs. If someone gets lucky, maybe google will return the result. I keep asking myself what's the point of closing a wiki? On Wed, Jul 23, 2014 at 7:40 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: It only takes a moment to ask to be added as a wiki

Re: Why is the cassandra documentation such poor quality?

contributing to cassandra community, I wouldn't have spent 2.5 months porting Hector to C# manually. Perhaps the real cause is that some committers can't empathise with others in the community? On Wed, Jul 23, 2014 at 8:22 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: All

Re: moving older tables from SSD to HDD?

Hi Kevin, This is something we do plan to support, but don't right now. You can see the discussion around this and related issues here https://issues.apache.org/jira/browse/CASSANDRA-5863 (although it may seem unrelated at first glance). On Mon, Aug 4, 2014 at 8:43 PM, Kevin Burton

Re: cassandra-stress with clustering columns?

The stress tool in 2.1 also now supports clustering columns: http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema There are however some features up for revision before release in order to help generate realistic workloads. See

Re: cassandra-stress with clustering columns?

using 2.0.9 unfortunately. :/ I will keep that in mind for when we upgrade. On Tue, Aug 19, 2014 at 10:51 AM, Benedict Elliott Smith belliottsm...@datastax.com wrote: The stress tool in 2.1 also now supports clustering columns: http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress

Re: Latest 2.1 and Datastax driver questions...

The run versions = x.x.6 is IMO an out of date trope. Not only does 2.1 have more than twice as many engineers working full time on it, and five times the QA engineers (which is cumulative with prior QA), we are also seeing many more users in the wild running release candidates and providing

Re: ETA for Cassandra 2.1 final release

It's up for vote right now, so should be a just few days unless something unexpected happens. On Mon, Sep 8, 2014 at 4:46 PM, Eugene Voytitsky viy@gmail.com wrote: Hi all, is there preliminary date when Cassandra 2.1 be finally released (not beta/rc)? -- Best regards, Eugene

Re: When CHANGES and JIRA Fix Versions disagree which should we believe?

In this case, it seems more likely CHANGES.txt will be correct, since it is maintained *at time of commit*, whereas JIRA fix versions can be forgotten to be maintained. On Tue, Sep 9, 2014 at 7:07 AM, Robert Coli rc...@eventbrite.com wrote: On Mon, Sep 8, 2014 at 2:56 PM, Peter Haggerty

Re: Performance testing in Cassandra

With the official release of 2.1, I highly recommend using the new stress tool bundled with it - it is improved in many ways over the tool in 2.0, and is compatible with older clusters. It supports the same simple mode of operation as the old stress, with better command line interface and more

Re: Mutation Stage does not finish

Could you post the results of jstack on the process somewhere? On Thu, Sep 11, 2014 at 7:07 AM, Robert Coli rc...@eventbrite.com wrote: On Wed, Sep 10, 2014 at 1:53 PM, Eduardo Cusa eduardo.c...@usmediaconsulting.com wrote: No, is still running the Mutation Stage. If you're sure that it

Re: no change observed in read latency after switching from EBS to SSD storage

It is possible this is CPU bound. In 2.1 we have optimised the comparison of clustering columns (CASSANDRA-5417 https://issues.apache.org/jira/browse/CASSANDRA-5417), but in 2.0 it is quite expensive. So for a large row with several million comparisons to perform (to merge, filter, etc.) it could

Re: how to force cassandra-stress to actually generate enough data

cassandra-stress has some (many) limitations - that I had planned to address now it's seeing wider adoption, but since I no longer work on the project for my day job I am unlikely to now... so, sorry but you'll have to tolerate them :) In particular, the problem you encounter here is that a given

Re: Cassandra Collections performance issue

If the overwrites are per map key there are no tombstones generated; only if the whole map is re-imaged are tombstones created, and prior to 3.0 this indeed can be major problem if done frequently. Prior to 3.0 collections also forbid certain optimisations to cell comparisons, and as a result can

Re: Flush activity and dropped messages

You should update from 2.0 to avoid this behaviour, is the simple answer. You are correct that when the commit log gets full the memtables are flushed to make room. 2.0 has several interrelated problems here though: There is a maximum flush queue length property (I cannot recall its name), and

Re: Guidelines for configuring Thresholds for Cassandra metrics

t is often not the ideal once you go through more thorough > testing, all of which I said initially and I still think is a reasonable > statement. > > -regards, > > Ryan Svihla > > > > > > > On Sat, Aug 27, 2016 at 9:31 AM -0500, "Benedict Elliott Smith"

Re: Partition size

On 12 September 2016 at 12:10, Mark Thomas <ma...@apache.org> wrote: > On 09/09/2016 21:11, Benedict Elliott Smith wrote: > > Come on. This kind of inconsistent 'policing' is not helpful. > > How is it inconsistent? Since I subscribed to the mailing list on 22 >

Re: Partition size

4:16, Mark Thomas <ma...@apache.org> wrote: > On 12/09/2016 12:51, Benedict Elliott Smith wrote: > > Please tone down your language. There is no need for profanity. > > Now is probably a good time to remind everyone of the Apache Code of > Conduct: > http://www.apache

Re: External linking Was: Isolation in case of Single Partition Writes and Batching with LWT

Again: *dev list*. "Please Note: These requirements apply to Apache projects: that is, to *individual committer and PMC behaviors* and actions within the context of collaboratively building software products at The Apache Software Foundation. By definition here, "Apache project" means the

Re: Partition size

Come on. This kind of inconsistent 'policing' is not helpful. By all means, push the *committers* to improve the project docs as is happening, and to promote the internal resources over external ones. But Mark has absolutely no formal connection with the project, and his contributions have only

Re: Isolation in case of Single Partition Writes and Batching with LWT

Yes, each partition modified by a batch has its modifications applied altogether, atomically (at the node level). On Friday, 9 September 2016, Bhuvan Rawal wrote: > As per this > doc > conditional batches

Re: Guidelines for configuring Thresholds for Cassandra metrics

as a blanket recommendation, > just a starting one. > > _ > From: Benedict Elliott Smith <bened...@apache.org > <javascript:_e(%7B%7D,'cvml','bened...@apache.org');>> > Sent: Friday, August 26, 2016 9:40 AM > Subject: Re: Guidelines for configuring Thres

Re: Guidelines for configuring Thresholds for Cassandra metrics

The default when I wrote it was 0.4 but it was found this did not saturate flush writers in JBOD configurations. Iirc it now defaults to 1/(1+#disks) which is not a terrible default, but obviously comes out much lower if you have many disks. This smaller value behaves better for peak performance,

Re: Cassandra data model right definition

Cassandra is not a "wide column store" anymore. It has a schema. Only thrift users no longer think they have a schema (though they do), and thrift is being deprecated. I really wish everyone would kill the term "wide column store" with fire. It seems to have never meant anything beyond

Re: Cassandra data model right definition

that describes a rowstore >>> >>> http://cs-www.cs.yale.edu/homes/dna/talks/abadi-sigmod08-slides.pdf >>> https://en.wikipedia.org/wiki/Column-oriented_DBMS#Row-oriented_systems >>> >>> Does it like this: >>> >>> 001:10,Smith,Joe,4; >

Re: Cassandra data model right definition

elational db could (and I'm sure one does!) allow for sparse fields >>> as well. MySQL can be backed by rocksdb now, does that make it not a row >>> store? >>> >>> You're arguing that everything is wrong but you're not proposing an >>> alternative,

Re: Cassandra data model right definition

gt; > If your so eager to deal with my pull request please review this one: > I would rather you review this pull request: https://issues. > apache.org/jira/browse/CASSANDRA-10825 > > > > > > On Mon, Oct 3, 2016 at 1:04 PM, Benedict Elliott Smith < > bened...@apache

Re: Cassandra data model right definition

still use. > On Fri, Sep 30, 2016 at 1:53 PM DuyHai Doan <doanduy...@gmail.com> wrote: > >> I used to present Cassandra as a NoSQL datastore with "distributed" >> table. This definition is closer to CQL and has some academic background >> (distributed hash

Re: JVM safepoints, mmap, and slow disks

it's still going to read quite a bit extra. > > Ariel > > On Sun, Oct 9, 2016, at 05:39 AM, Benedict Elliott Smith wrote: > > The biggest problem with pread was the issue of over reading (reading 64k > where 4k would suffice), which was significantly improved in 2.2 iirc. I

Re: How does the "batch" commit log sync works

That is the maximum length of time that queries may be batched together for, not the minimum. If there is a break in the flow of queries for the commit log, it will commit those outstanding immediately. It will anyway commit in clusters of commit log file size (default 32Mb). I know the

Re: JVM safepoints, mmap, and slow disks

The biggest problem with pread was the issue of over reading (reading 64k where 4k would suffice), which was significantly improved in 2.2 iirc. I don't think the penalty is very significant anymore, and if you are experiencing time to safe point issues it's very likely a worthwhile switch to