Re: Cassandra crashes....

Yep, similar symptoms - but no, there's no OOM killer Also, if you look in the gc log around the time of failure, the heap memory was much below the 16 GB limit. And if I look at the 2nd last GC log before the crash, here’s what we see. And you will notice that cleaning up the 4 GB Eden (along wi

Re: Cassandra crashes....

So the reason for the large number of prepared statements is because of the nature of the application. One of the periodic job does lookup with a partial key (key prefix, not filtered queries) for thousands of rows. Hence the large number of prepared statements. Almost of the queries once execut

Re: Cassandra isn't compacting old files

I issued another major compaction just now and a brand new SSTable in Level 2 has an Estimated droppable tombstone value of 0.64. I don't know how accurate that is. On Tuesday, August 22, 2017, 9:33:34 PM PDT, Sotirios Delimanolis wrote: What do you mean by "a single SSTable"? SSTable size is

Re: Cassandra isn't compacting old files

What do you mean by "a single SSTable"? SSTable size is set to 200MB and there are ~ 100 SSTables in that previous example in Level 3. This previous example table doesn't have a TTL, but we do delete rows. I've since compacted the table so I can't provide the previous "Estimated droppable tombst

Re: Cassandra isn't compacting old files

LCS major compaction on 2.2 should compact each level to have a single SSTable. It seems more likely to me that you are simply not generating enough data to require compactions in L3 and most data is TTL'ing before it gets there. Out of curiosity, what does sstablemetadata report for Estimated dro

Re: Bootstrapping a node fails because of compactions not keeping up

What version are you running? 2.2 has an improvement that will retain levels when streaming and this shouldn't really happen. If you're on 2.1 best bet is to upgrade

Re: Cassandra crashes....

sounds like Cassandra is being killed by the oom killer. can you check dmesg to see if this is the case? sounds a bit absurd with 256g of memory but could be a config problem.

Re: ExceptionInInitializerError encountered during startup

Reporting back... I gave up and did this work in a stand-alone project where /EmbeddedCass//andraServ//erHelper.startEmbeddedCassandra()/ works fine. I think now that Cassandra's dependency upon /slf4j/ clashes with what we've had to do in our greater product to regulate which version of /slf

Re: Cassandra crashes....

On 08/22/2017 05:39 PM, Thakrar, Jayesh wrote: Surbhi and Fay, I agree we have plenty of RAM to spare. Hi At the very beginning of system.log there is a INFO [CompactionExecutor:487] 2017-08-21 23:21:01,684 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate

Re: Upgrade requirements for upgrading from cassandra 2.1.x to 2.2.x

NEWS.txt is the goto spot for upgrade instructions, caveats, etc. Jon > On Aug 22, 2017, at 2:46 PM, Chuck Reynolds wrote: > > Anyone? > > From: "Chuck (me) Reynolds" > Reply-To: "user@cassandra.apache.org" > Date: Tuesday, August 22, 2017 at 9:40 AM > To: "user@cassandra.apache.org" > Sub

Re: Upgrade requirements for upgrading from cassandra 2.1.x to 2.2.x

Anyone? From: "Chuck (me) Reynolds" Reply-To: "user@cassandra.apache.org" Date: Tuesday, August 22, 2017 at 9:40 AM To: "user@cassandra.apache.org" Subject: Upgrade requirements for upgrading from cassandra 2.1.x to 2.2.x Where can I find requirements to upgrade from Cassandra 2.1.x to 2.2.x?

Re: ExceptionInInitializerError encountered during startup

Thanks, Myrle. This confirms what I've tried so far. The problem may be an assumed requirement, such as the YAML file and perhaps /log4j-embedded-cassandra.properties/. But, I'm supplying both of those. This has something to do with /slf4j/ logging, a logger that has no name when it goes to get

Re: Cassandra isn't compacting old files

Ignore the files missing those other components, that was confirmation bias :( I was sorting by date instead of by name and just assumed that something was wrong with Cassandra. Here's an example table's SSTables, sorted by level, then by repaired status: SSTable [name=lb-432055-big-Data.db, leve

Re: Limit on having number of nodes in C* cluster

You can't, you typically need to add a whole new datacenter, replicate your data and traffic there, and decommission the old one -- Jeff Jirsa > On Aug 22, 2017, at 8:56 AM, techpyaasa . wrote: > > How can I decrease tokens for existing nodes? > Doesn't it create problem? > > > On Aug 22,

Re: Limit on having number of nodes in C* cluster

How can I decrease tokens for existing nodes? Doesn't it create problem? On Aug 22, 2017 7:22 PM, "Vladimir Yudovin" wrote: Probably decreasing tokens number can help to mange big cluster? Best regards, Vladimir Yudovin, *Winguzone

Re: Cassandra crashes....

We are using TWCS compaction. Here's one sample table CREATE TABLE ae.raw_logs_by_user ( dtm_id bigint, company_id int, source text, status_id int, log_date bigint, uuid_least bigint, uuid_most bigint, profile_system_id int, parent_message_id int, parent_te

Re: Cassandra crashes....

what kind compaction? LCS ? On Aug 22, 2017 8:39 AM, "Thakrar, Jayesh" wrote: Surbhi and Fay, I agree we have plenty of RAM to spare. However, our data load and compaction churn is so high (partially thanks to SSDs!), its causing too much GC pressure. And as you know the Edenspace and survi

Upgrade requirements for upgrading from cassandra 2.1.x to 2.2.x

Where can I find requirements to upgrade from Cassandra 2.1.x to 2.2.x? I would like to know things like do I have to do an SStable upgrade or not. Thanks

Re: Cassandra crashes....

16GB heap is too small for G1GC . Try at least 32GB of heap size On Tue, Aug 22, 2017 at 7:58 AM Fay Hou [Storage Service] ­ < fay...@coupang.com> wrote: > What errors do you see? > 16gb of 256 GB . Heap is too small. I would give heap at least 160gb. > > > On Aug 22, 2017 7:42 AM, "Thakrar, Jayes

Re: Cassandra crashes....

What errors do you see? 16gb of 256 GB . Heap is too small. I would give heap at least 160gb. On Aug 22, 2017 7:42 AM, "Thakrar, Jayesh" wrote: Hi All, We are somewhat new users to Cassandra 3.10 on Linux and wanted to ping the user group for their experiences. Our usage profile is batch

Re: Cassandra crashes....

You typically don't want to set the eden space when you're using G1 -- Jeff Jirsa > On Aug 22, 2017, at 7:42 AM, Thakrar, Jayesh > wrote: > > Hi All, > > We are somewhat new users to Cassandra 3.10 on Linux and wanted to ping the > user group for their experiences. > > Our usage profile

Re: ExceptionInInitializerError encountered during startup

On Tue, Aug 22, 2017 at 4:21 PM, Russell Bateman wrote: > As this was my first post to this forum, I wonder if someone would reply to > it if only to prove to myself that I've not posted to /dev/null as it were > even if there's no answer or the question is stupid, etc. (Note: I am > getting other

Cassandra crashes....

Hi All, We are somewhat new users to Cassandra 3.10 on Linux and wanted to ping the user group for their experiences. Our usage profile is batch jobs that load millions of rows to Cassandra every hour. And there are similar period batch jobs that read millions of rows and do some processing,

Re: ExceptionInInitializerError encountered during startup

As this was my first post to this forum, I wonder if someone would reply to it if only to prove to myself that I've not posted to //dev/null/ as it were even if there's no answer or the question is stupid, etc. (Note: I am getting other forum posts, but maybe what I've posted didn't reach the f

Re: Limit on having number of nodes in C* cluster

Probably decreasing tokens number can help to mange big cluster? Best regards, Vladimir Yudovin, Winguzone - Cloud Cassandra Hosting On Mon, 21 Aug 2017 19:38:37 -0400 Eduard Tudenhoefner wrote We've been doing successful testing with mul

Bootstrapping a node fails because of compactions not keeping up

Hi all, I am trying to bootstrap a node without success due to running out of space. Average node size is 260GB with lots of LCS tables (overall data ~3.5 TB) Each node is also configured with a 1TB disk, including the bootstrapping node. After 12 hours the bootstrapping node fails with more than

Commit log archive configuration

Hello there, in https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configLogArchive.html it is mentioned that: “The commit log is archived at node startup and when a commit log is written to disk, or at a specified point-in-time” How do I configure the archive mechanism to run

Commit log archiver

Hello there, I’m running cassandra 3.11.0 and I have configured commitlog_archiving.properties to run a script. That script rsyncs the commit log files to a backup server. If the rsync is successful then the script exits with status 0 and the commit log file is deleted from the cassandra node.