Re: Migrating Cassandra from 3.11.11 to 4.0.0 vs num_tokens

[Jean Tremblay]

Great Thank you for the answer and the link!


> On 4 Sep 2021, at 11:35, Erick Ramirez  wrote:
> 
> It isn't possible to change the tokens on a node once it is already part of 
> the cluster. Cassandra won't allow you to do it because it will make the data 
>  already on disk unreadable. You'll need to either configure new nodes or add 
> a new DC. I've answered an identical question in 
> https://community.datastax.com/questions/12213/ 
>  where I've provided steps 
> for the 2 options. I hope to draft a runbook and get it published on the 
> Apache website in the coming days. Cheers!

Migrating Cassandra from 3.11.11 to 4.0.0 vs num_tokens

[Jean Tremblay]

Hi,

We are currently running Cassandra 3.11.11 with the default values for 
num_tokens: 256.
We want to migrate to Cassandra 4.0.0 which has default values for num_tokens 
set to 16.

Is it safe to migrate with the default values, i.e. can I leave it set to 16 
when migrating to Cassandra 4.0.0 or is there something special which needs to 
be done?

Thanks for your help.

Cheers

Jean

Re: COPY command with where condition

[Jean Tremblay]

Did you think about using a Materialised View to generate what you want to 
keep, and then use DSBulk to extract the data?

> On 17 Jan 2020, at 14:30 , adrien ruffie  wrote:
> 
> Sorry I come back to a quick question about the bulk loader ...
> 
> https://www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader 
> 
> 
> I read this : "Operations such as converting strings to lowercase, arithmetic 
> on input columns, or filtering out rows based on some criteria, are not 
> supported. "
> 
> Consequently, it's still not possible to use a WHERE clause with DSBulk, 
> right ?
> 
> I don't really know how I can do it, in order to don't keep the wholeness of 
> business data already stored and which don't need to export...
> 
> 
> 
> De : adrien ruffie 
> Envoyé : vendredi 17 janvier 2020 11:39
> À : Erick Ramirez ; user@cassandra.apache.org 
> 
> Objet : RE: COPY command with where condition
>  
> Thank a lot !
> It's a good news for DSBulk ! I will take a look around this solution.
> 
> best regards,
> Adrian
> De : Erick Ramirez 
> Envoyé : vendredi 17 janvier 2020 10:02
> À : user@cassandra.apache.org 
> Objet : Re: COPY command with where condition
>  
> The COPY command doesn't support filtering and it doesn't perform well for 
> large tables.
> 
> Have you considered the DSBulk tool from DataStax? Previously, it only worked 
> with DataStax Enterprise but a few weeks ago, it was made free and works with 
> open-source Apache Cassandra. For details, see this blogpost 
> . Cheers!
> 
> On Fri, Jan 17, 2020 at 6:57 PM adrien ruffie  > wrote:
> Hello all,
> 
> In my company we want to export a big dataset of our cassandra's ring.
> We search to use COPY command but I don't find if and how can a WHERE 
> condition can be use ?
> 
> Because we need to export only several data which must be return by a WHERE 
> closure, specially
> and unfortunately with ALLOW FILTERING due to several old tables which were 
> poorly conceptualized...
> 
> Do you know a means to do that please ?
> 
> Thank all and best regards
> 
> Adrian   



smime.p7s
Description: S/MIME cryptographic signature

Re: sstableloader

[Jean Tremblay]

Thank you for your answer Kai.

On 17 Aug 2016, at 11:34 , Kai Wang <dep...@gmail.com<mailto:dep...@gmail.com>> 
wrote:

yes, you are correct.

On Tue, Aug 16, 2016 at 2:37 PM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Hi,

I’m using Cassandra 3.7.

In the documentation for sstableloader I read the following:

<< Note: To get the best throughput from SSTable loading, you can use multiple 
instances of sstableloader to stream across multiple machines. No hard limit 
exists on the number of SSTables that sstableloader can run at the same time, 
so you can add additional loaders until you see no further improvement.>>

Does this mean that I can stream my sstables to my cluster from many instance 
of sstableloader running simultaneously on many client machines?

I ask because I would like to improve the transfer speed of my stables to my 
cluster.

Kind regards and thanks for your comments.

Jean

Re: Most stable version?

[Jean Tremblay]

Thank you Jack.
Jean
On 14 Apr 2016, at 22:00 , Jack Krupansky 
<jack.krupan...@gmail.com<mailto:jack.krupan...@gmail.com>> wrote:

Normally, since 3.5 just came out, it would be wise to see if people report any 
problems over the next few weeks.

But... the new tick-tock release process is designed to assure that these 
odd-numbered releases are only incremental bug fixes from the last 
even-numbered feature release, which was 3.4. So, 3.5 should be reasonably 
stable.

That said, a bug-fix release of 3.0 is probably going to be more stable than a 
bug fix release of a more recent feature release (3.4).

Usually it comes down to whether you need any of the new features or 
improvements in 3.x, or whether you might want to keep your chosen release in 
production for longer than the older 3.0 releases will be in production.

Ultimately, this is a personality test: Are you adventuresome or conservative?

To be clear, with the new tick-tock release scheme, 3.5 is designed to be a 
stable release.

-- Jack Krupansky

On Thu, Apr 14, 2016 at 3:23 PM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Hi,
Could someone give his opinion on this?
What should be considered more stable, Cassandra 3.0.5 or Cassandra 3.5?

Thank you
Jean

> On 12 Apr,2016, at 07:00, Jean Tremblay 
> <jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
> wrote:
>
> Hi,
> Which version of Cassandra should considered most stable in the version 3?
> I see two main branch: the branch with the version 3.0.* and the tick-tock 
> one 3.*.*.
> So basically my question is: which one is most stable, version 3.0.5 or 
> version 3.3?
> I know odd versions in tick-took are bug fix.
> Thanks
> Jean

Re: Most stable version?

[Jean Tremblay]

Hi,
Could someone give his opinion on this?
What should be considered more stable, Cassandra 3.0.5 or Cassandra 3.5?

Thank you
Jean

> On 12 Apr,2016, at 07:00, Jean Tremblay <jean.tremb...@zen-innovations.com> 
> wrote:
> 
> Hi,
> Which version of Cassandra should considered most stable in the version 3?
> I see two main branch: the branch with the version 3.0.* and the tick-tock 
> one 3.*.*.
> So basically my question is: which one is most stable, version 3.0.5 or 
> version 3.3?
> I know odd versions in tick-took are bug fix. 
> Thanks
> Jean

Most stable version?

[Jean Tremblay]

Hi,
Which version of Cassandra should considered most stable in the version 3?
I see two main branch: the branch with the version 3.0.* and the tick-tock one 
3.*.*.
So basically my question is: which one is most stable, version 3.0.5 or version 
3.3?
I know odd versions in tick-took are bug fix. 
Thanks
Jean

Re: Large number of tombstones without delete or update

[Jean Tremblay]

Ralf,

Thank YOU very much Ralf. You are the first one who could finally shed some 
light on something I observed, but I could not put my finger on what exactly is 
causing my Tombstones.
I cannot judge your method for evaluating the amount of tombstone. It seems 
valid to me.

Jean
On 24 Mar 2016, at 11:10 , Ralf Steppacher 
<ralf.viva...@gmail.com<mailto:ralf.viva...@gmail.com>> wrote:

Jean,

yes, I am using the native protocol v4 (auto-negotiated between java driver 
3.0.0 and C* 2.2.4, verified by logging 
cluster.getConfiguration().getProtocolOptions().getProtocolVersion() ).

My first approach for testing for tombstones was “brute force”. Add records and 
soon enough (after about 2000 records were inserted) a query for the row count 
would yield warnings in the C* log:

WARN  [SharedPool-Worker-2] 2016-03-23 16:54:43,134 SliceQueryFilter.java:307 - 
Read 2410 live and 4820 tombstone cells in 
event_log_system.event_by_patient_timestamp for key: 100013866046895035 (see 
tombstone_warn_threshold). 5000 columns were requested, slices=[2040-06-02 
05\:57+0200:!-]



I then added query trace logging for the count query. I dropped the whole 
keyspace, inserted a single JSON message and then issued the count query:

Host (queried): velcassandra/10.211.55.8:9042
Host (tried): velcassandra/10.211.55.8:9042
Trace id: de69df90-f1a0-11e5-a558-f3993541f01b

activity   | timestamp| source | 
source_elapsed
---+--++--
Executing single-partition query on event_by_patient_timestamp | 10:14:53.324 | 
/127.0.0.1 | 2815
  Acquiring sstable references | 10:14:53.324 | /127.0.0.1 | 
3036
   Merging memtable tombstones | 10:14:53.324 | /127.0.0.1 | 
3097
Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones | 
10:14:53.324 | /127.0.0.1 | 3153
Merging data from memtables and 0 sstables | 10:14:53.324 | /127.0.0.1 |
 3170
 Read 1 live and 0 tombstone cells | 10:14:53.324 | /127.0.0.1 | 
3202

Note the last line states that 0 tombstone cells were read. That was after I 
made sure I had no null text fields in the JSON message. With missing/null JSON 
fields (mapped to columns of type text) the trace always reported >= 1 read 
tombstone cells.

I used the version yielding 0 read tombstones again in the brute force test and 
Cassandra never logged any warnings.

Is this a valid test?


Ralf


On 24.03.2016, at 10:46, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:

Ralf,

Are you using protocol V4?
How do you measure if a tombstone was generated?

Thanks

Jean

Re: Large number of tombstones without delete or update

[Jean Tremblay]

Ralf,

Are you using protocol V4?
How do you measure if a tombstone was generated?

Thanks

Jean

On 24 Mar 2016, at 10:35 , Ralf Steppacher 
> wrote:

How does this improvement apply to inserting JSON? The prepared statement has 
exactly one parameter and it is always bound to the JSON message:

INSERT INTO event_by_patient_timestamp JSON ?

How would I “unset” a field inside the JSON message written to the 
event_by_patient_timestamp table?


Ralf


On 24.03.2016, at 10:22, Peer, Oded 
> wrote:

http://www.datastax.com/dev/blog/datastax-java-driver-3-0-0-released#unset-values

“For Protocol V3 or below, all variables in a statement must be bound. With 
Protocol V4, variables can be left “unset”, in which case they will be ignored 
server-side (no tombstones will be generated).”


From: Ralf Steppacher [mailto:ralf.viva...@gmail.com]
Sent: Thursday, March 24, 2016 11:19 AM
To: user@cassandra.apache.org
Subject: Re: Large number of tombstones without delete or update

I did some more tests with my particular schema/message structure:

A null text field inside a UDT instance does NOT yield tombstones.
A null map does NOT yield tombstones.
A null text field does yield tombstones.


Ralf

On 24.03.2016, at 09:42, Ralf Steppacher 
> wrote:

I can confirm that if I send JSON messages that always cover all schema fields 
the tombstone issue is not reported by Cassandra.
So, is there a way to work around this issue other than to always populate 
every column of the schema with every insert? That would be a pain in the 
backside, really.

Why would C* not warn about the excessive number of tombstones if queried from 
cqlsh?


Thanks!
Ralf



On 23.03.2016, at 19:09, Robert Coli 
> wrote:

On Wed, Mar 23, 2016 at 9:50 AM, Ralf Steppacher 
> wrote:
How come I end up with that large a number of tombstones?

Are you inserting NULLs?

=Rob

Re: Large number of tombstones without delete or update

[Jean Tremblay]

Same for me. Only inserts not replacing old records.

On 24 Mar,2016, at 07:42, Ralf Steppacher 
> wrote:

Eric,

I am writing the whole record in a single INSERT INTO ... JSON. I am not 
"insert-updating" over an existing record nor do I run any UPDATE statements.


Ralf

On 24.03.2016, at 01:47, Eric Stevens 
> wrote:

In addition to writing null values acting as tombstones, also INSERTing a 
collection (or UPDATE where you set the collection rather than append to it) 
are also operations which will create tombstones.

On Wed, Mar 23, 2016 at 12:09 PM Robert Coli 
> wrote:
On Wed, Mar 23, 2016 at 9:50 AM, Ralf Steppacher 
> wrote:
How come I end up with that large a number of tombstones?

Are you inserting NULLs?

=Rob

Re: Large number of tombstones without delete or update

[Jean Tremblay]

Hi,

I also have loads of tombstones while only inserting new rows without ever 
deleting rows.

My rows contain null columns and also collections.

How can I avoid the creation of these tombstones?

Thanks for your help.

On 24 Mar,2016, at 02:08, Steve Robenalt 
> wrote:

Hi Henry,

Since new values are written without checking for previous values, the only way 
to assure that a nulled column blocks a previously valid one is to write a 
tombstone to occlude whatever value might have been present. I believe there 
are some recent changes as to the handling of null values to relax some of this 
behavior under some circumstances, but I'm not sure as to which version it 
affects or what the circumstances are. A similar question was raised recently 
on this list, so the list archives may be of some use.

Steve


On Wed, Mar 23, 2016 at 6:00 PM, Henry M 
> wrote:
What is the reason for the tombstone for a brand new insert? Do the fields get 
written as a whole (both nulls and non-nulls?

I understand the rationale for tombstones for deletes and updates but it does 
not make sense for an insert (I am trying to make sense of it). I understand 
Cassandra writes the record without first checking its existence but wouldn't 
the whole set of fields including values to null out be applied as one single 
operation?

Thank you,
Henry



On Wed, Mar 23, 2016 at 5:47 PM, Eric Stevens 
> wrote:
In addition to writing null values acting as tombstones, also INSERTing a 
collection (or UPDATE where you set the collection rather than append to it) 
are also operations which will create tombstones.

On Wed, Mar 23, 2016 at 12:09 PM Robert Coli 
> wrote:
On Wed, Mar 23, 2016 at 9:50 AM, Ralf Steppacher 
> wrote:
How come I end up with that large a number of tombstones?

Are you inserting NULLs?

=Rob





--
Steve Robenalt
Software Architect
sroben...@highwire.org
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Re: Rename Keyspace offline

[Jean Tremblay]

Thank you all for your replies.
My main objective was not to change my client.
After your answers it makes a lot of sense to modify my client in a way to make 
it accept different key space name. This way I will no longer need to rename a 
key space I simply need to develop a way to tell my client that there is a new 
key space.

Thanks again for your feedback
Jean

On 27 Jan,2016, at 19:58, Robert Coli 
<rc...@eventbrite.com<mailto:rc...@eventbrite.com>> wrote:

On Wed, Jan 27, 2016 at 6:49 AM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Since it takes me 2 days to load my data, I was planning to load the new set on 
a new keyspace (KS-Y), and when loaded drop KS-X and rename KS-Y to KS-X.

Why bother with the rename? Just have two keyspaces, foo and foo_, and 
alternate your bulk loads between truncating them?

Would this procedure work to destroy an old keyspace KS-X and rename a new 
keyspace KS-Y to KS-X:

Yes, if you include :

0) Load schema for KS-Y into KS-X

1) nodetool drain each node.
2) stop cassandra on each node.
3) on each node:
3.1) rm -r data/KS-X
3.2) mv data/KS-Y data/KS-X
4) restart each node.

Note also that in step 3.2, the uuid component of file and/or directory names 
will have to be changed.

=Rob

Rename Keyspace offline

[Jean Tremblay]

Hi,

I have a huge set of data, which takes about 2 days to bulk load on a Cassandra 
3.0 cluster of 5 nodes. That is about 13 billion rows.

Quite often I need to reload this data, new structure, or data is reorganise. 
There are clients reading from a given keyspace (KS-X).

Since it takes me 2 days to load my data, I was planning to load the new set on 
a new keyspace (KS-Y), and when loaded drop KS-X and rename KS-Y to KS-X.

Now I know "renaming keyspace" is a functionality which was removed.

Would this procedure work to destroy an old keyspace KS-X and rename a new 
keyspace KS-Y to KS-X:

1) nodetool drain each node.
2) stop cassandra on each node.
3) on each node:
3.1) rm -r data/KS-X
3.2) mv data/KS-Y data/KS-X
4) restart each node.

Could someone please confirm this? I guess it would work, but I’m just afraid 
that there could be in some system table some information that would not allow 
this.

Thanks for your help.

Cheers

Jean

Re: Cassandra 3.1.1 with respect to HeapSpace

[Jean Tremblay]

Thank you Sebastián!

On 15 Jan 2016, at 19:09 , Sebastian Estevez 
<sebastian.este...@datastax.com<mailto:sebastian.este...@datastax.com>> wrote:

The recommended (and default when available) heap size for Cassandra is 8GB and 
for New size it's 100mb per core.

Your milage may vary based on workload, hardware etc.

There are also some alternative JVM tuning schools of thought. See 
cassandra-8150 (large heap) and CASSANDRA-7486 (G1GC).



All the best,

[datastax_logo.png]<http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | 
sebastian.este...@datastax.com<mailto:sebastian.este...@datastax.com>
[linkedin.png]<https://www.linkedin.com/company/datastax> [facebook.png] 
<https://www.facebook.com/datastax>  [twitter.png] 
<https://twitter.com/datastax>  [g+.png] 
<https://plus.google.com/+Datastax/about>  
[https://lh6.googleusercontent.com/24_538J0j5M0NHQx-jkRiV_IHrhsh-98hpi--Qz9b0-I4llvWuYI6LgiVJsul0AhxL0gMTOHgw3G0SvIXaT2C7fsKKa_DdQ2uOJ-bQ6h_mQ7k7iMybcR1dr1VhWgLMxcmg]
 <http://feeds.feedburner.com/datastax>
<http://goog_410786983/>

[http://learn.datastax.com/rs/059-YLZ-577/images/Gartner_728x90_Sig4.png]<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology, 
delivering Apache Cassandra to the world’s most innovative enterprises. 
Datastax is built to be agile, always-on, and predictably scalable to any size. 
With more than 500 customers in 45 countries, DataStax is the database 
technology and transactional backbone of choice for the worlds most innovative 
companies such as Netflix, Adobe, Intuit, and eBay.

On Fri, Jan 15, 2016 at 4:00 AM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Thank you Sebastián for your useful advice. I managed restarting the nodes, but 
I needed to delete all the commit logs, not only the last one specified. 
Nevertheless I’m back in business.

Would there be a better memory configuration to select for my nodes in a C* 3 
cluster? Currently I use MAX_HEAP_SIZE=“6G" HEAP_NEWSIZE=“496M” for a 16M RAM 
node.

Thanks for your help.

Jean

On 15 Jan 2016, at 24:24 , Sebastian Estevez 
<sebastian.este...@datastax.com<mailto:sebastian.este...@datastax.com>> wrote:

Try starting the other nodes. You may have to delete or mv the commitlog 
segment referenced in the error message for the node to come up since 
apparently it is corrupted.

All the best,

[datastax_logo.png]<http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615<tel:954%20905%208615> | 
sebastian.este...@datastax.com<mailto:sebastian.este...@datastax.com>
[linkedin.png]<https://www.linkedin.com/company/datastax> [facebook.png] 
<https://www.facebook.com/datastax>  [twitter.png] 
<https://twitter.com/datastax>  [g+.png] 
<https://plus.google.com/+Datastax/about>  
[https://lh6.googleusercontent.com/24_538J0j5M0NHQx-jkRiV_IHrhsh-98hpi--Qz9b0-I4llvWuYI6LgiVJsul0AhxL0gMTOHgw3G0SvIXaT2C7fsKKa_DdQ2uOJ-bQ6h_mQ7k7iMybcR1dr1VhWgLMxcmg]
 <http://feeds.feedburner.com/datastax>
<http://goog_410786983/>

[http://learn.datastax.com/rs/059-YLZ-577/images/Gartner_728x90_Sig4.png]<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology, 
delivering Apache Cassandra to the world’s most innovative enterprises. 
Datastax is built to be agile, always-on, and predictably scalable to any size. 
With more than 500 customers in 45 countries, DataStax is the database 
technology and transactional backbone of choice for the worlds most innovative 
companies such as Netflix, Adobe, Intuit, and eBay.

On Thu, Jan 14, 2016 at 1:00 PM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
How can I restart?
It blocks with the error listed below.
Are my memory settings good for my configuration?

On 14 Jan 2016, at 18:30, Jake Luciani 
<jak...@gmail.com<mailto:jak...@gmail.com>> wrote:

Yes you can restart without data loss.

Can you please include info about how much data you have loaded per node and 
perhaps what your schema looks like?

Thanks

On Thu, Jan 14, 2016 at 12:24 PM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:

Ok, I will open a ticket.

How could I restart my cluster without loosing everything ?
Would there be a better memory configuration to select for my nodes? Currently 
I use MAX_HEAP_SIZE="6G" HEAP_NEWSIZE=“496M” for a 16M RAM node.

Thanks

Jean

On 14 Jan 2016, at 18:19, Tyler Hobbs 
<ty...@datastax.com<mailto:ty...@datastax.com>> wrote:

I don't think that's a known issue.  Can you open a ticket at 
https://issues.apache.org/jira/browse/CASSANDRA and attach your schema al

Re: Cassandra 3.1.1 with respect to HeapSpace

[Jean Tremblay]

Thank you Sebastián for your useful advice. I managed restarting the nodes, but 
I needed to delete all the commit logs, not only the last one specified. 
Nevertheless I’m back in business.

Would there be a better memory configuration to select for my nodes in a C* 3 
cluster? Currently I use MAX_HEAP_SIZE=“6G" HEAP_NEWSIZE=“496M” for a 16M RAM 
node.

Thanks for your help.

Jean

On 15 Jan 2016, at 24:24 , Sebastian Estevez 
<sebastian.este...@datastax.com<mailto:sebastian.este...@datastax.com>> wrote:

Try starting the other nodes. You may have to delete or mv the commitlog 
segment referenced in the error message for the node to come up since 
apparently it is corrupted.

All the best,

[datastax_logo.png]<http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | 
sebastian.este...@datastax.com<mailto:sebastian.este...@datastax.com>
[linkedin.png]<https://www.linkedin.com/company/datastax> [facebook.png] 
<https://www.facebook.com/datastax>  [twitter.png] 
<https://twitter.com/datastax>  [g+.png] 
<https://plus.google.com/+Datastax/about>  
[https://lh6.googleusercontent.com/24_538J0j5M0NHQx-jkRiV_IHrhsh-98hpi--Qz9b0-I4llvWuYI6LgiVJsul0AhxL0gMTOHgw3G0SvIXaT2C7fsKKa_DdQ2uOJ-bQ6h_mQ7k7iMybcR1dr1VhWgLMxcmg]
 <http://feeds.feedburner.com/datastax>
<http://goog_410786983/>

[http://learn.datastax.com/rs/059-YLZ-577/images/Gartner_728x90_Sig4.png]<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology, 
delivering Apache Cassandra to the world’s most innovative enterprises. 
Datastax is built to be agile, always-on, and predictably scalable to any size. 
With more than 500 customers in 45 countries, DataStax is the database 
technology and transactional backbone of choice for the worlds most innovative 
companies such as Netflix, Adobe, Intuit, and eBay.

On Thu, Jan 14, 2016 at 1:00 PM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
How can I restart?
It blocks with the error listed below.
Are my memory settings good for my configuration?

On 14 Jan 2016, at 18:30, Jake Luciani 
<jak...@gmail.com<mailto:jak...@gmail.com>> wrote:

Yes you can restart without data loss.

Can you please include info about how much data you have loaded per node and 
perhaps what your schema looks like?

Thanks

On Thu, Jan 14, 2016 at 12:24 PM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:

Ok, I will open a ticket.

How could I restart my cluster without loosing everything ?
Would there be a better memory configuration to select for my nodes? Currently 
I use MAX_HEAP_SIZE="6G" HEAP_NEWSIZE=“496M” for a 16M RAM node.

Thanks

Jean

On 14 Jan 2016, at 18:19, Tyler Hobbs 
<ty...@datastax.com<mailto:ty...@datastax.com>> wrote:

I don't think that's a known issue.  Can you open a ticket at 
https://issues.apache.org/jira/browse/CASSANDRA and attach your schema along 
with the commitlog files and the mutation that was saved to /tmp?

On Thu, Jan 14, 2016 at 10:56 AM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Hi,

I have a small Cassandra Cluster with 5 nodes, having 16MB of RAM.
I use Cassandra 3.1.1.
I use the following setup for the memory:
  MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="496M"

I have been loading a lot of data in this cluster over the last 24 hours. The 
system behaved I think very nicely. It was loading very fast, and giving 
excellent read time. There was no error messages until this one:


ERROR [SharedPool-Worker-35] 2016-01-14 17:05:23,602 
JVMStabilityInspector.java:139 - JVM state determined to be unstable.  Exiting 
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_65]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_65]
at 
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:263)
 ~[apa

Re: Cassandra 3.1.1 with respect to HeapSpace

[Jean Tremblay]

How can I restart?
It blocks with the error listed below.
Are my memory settings good for my configuration?

On 14 Jan 2016, at 18:30, Jake Luciani 
<jak...@gmail.com<mailto:jak...@gmail.com>> wrote:

Yes you can restart without data loss.

Can you please include info about how much data you have loaded per node and 
perhaps what your schema looks like?

Thanks

On Thu, Jan 14, 2016 at 12:24 PM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:

Ok, I will open a ticket.

How could I restart my cluster without loosing everything ?
Would there be a better memory configuration to select for my nodes? Currently 
I use MAX_HEAP_SIZE="6G" HEAP_NEWSIZE=“496M” for a 16M RAM node.

Thanks

Jean

On 14 Jan 2016, at 18:19, Tyler Hobbs 
<ty...@datastax.com<mailto:ty...@datastax.com>> wrote:

I don't think that's a known issue.  Can you open a ticket at 
https://issues.apache.org/jira/browse/CASSANDRA and attach your schema along 
with the commitlog files and the mutation that was saved to /tmp?

On Thu, Jan 14, 2016 at 10:56 AM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Hi,

I have a small Cassandra Cluster with 5 nodes, having 16MB of RAM.
I use Cassandra 3.1.1.
I use the following setup for the memory:
  MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="496M"

I have been loading a lot of data in this cluster over the last 24 hours. The 
system behaved I think very nicely. It was loading very fast, and giving 
excellent read time. There was no error messages until this one:


ERROR [SharedPool-Worker-35] 2016-01-14 17:05:23,602 
JVMStabilityInspector.java:139 - JVM state determined to be unstable.  Exiting 
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_65]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_65]
at 
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:263)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:183)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:96)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:128)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_65]
at 
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
 ~[apac

Re: Cassandra 3.1.1 with respect to HeapSpace

[Jean Tremblay]

Ok, I will open a ticket.

How could I restart my cluster without loosing everything ?
Would there be a better memory configuration to select for my nodes? Currently 
I use MAX_HEAP_SIZE="6G" HEAP_NEWSIZE=“496M” for a 16M RAM node.

Thanks

Jean

On 14 Jan 2016, at 18:19, Tyler Hobbs 
<ty...@datastax.com<mailto:ty...@datastax.com>> wrote:

I don't think that's a known issue.  Can you open a ticket at 
https://issues.apache.org/jira/browse/CASSANDRA and attach your schema along 
with the commitlog files and the mutation that was saved to /tmp?

On Thu, Jan 14, 2016 at 10:56 AM, Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Hi,

I have a small Cassandra Cluster with 5 nodes, having 16MB of RAM.
I use Cassandra 3.1.1.
I use the following setup for the memory:
  MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="496M"

I have been loading a lot of data in this cluster over the last 24 hours. The 
system behaved I think very nicely. It was loading very fast, and giving 
excellent read time. There was no error messages until this one:


ERROR [SharedPool-Worker-35] 2016-01-14 17:05:23,602 
JVMStabilityInspector.java:139 - JVM state determined to be unstable.  Exiting 
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_65]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_65]
at 
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:263)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:183)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:96)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:128)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_65]
at 
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
[apache-cassandra-3.1.1.jar:3.1.1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

4 nodes out of 5 crashed with this error message. Now when I want to restart 
the first node I have the following error;

ERROR [main] 2016-01-14 17:15:59,617 JVMStabilityInspector.java:81 - Exiting 
due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayExce

Cassandra 3.1.1 with respect to HeapSpace

[Jean Tremblay]

Hi,

I have a small Cassandra Cluster with 5 nodes, having 16MB of RAM.
I use Cassandra 3.1.1.
I use the following setup for the memory:
  MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="496M"

I have been loading a lot of data in this cluster over the last 24 hours. The 
system behaved I think very nicely. It was loading very fast, and giving 
excellent read time. There was no error messages until this one:


ERROR [SharedPool-Worker-35] 2016-01-14 17:05:23,602 
JVMStabilityInspector.java:139 - JVM state determined to be unstable.  Exiting 
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_65]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_65]
at 
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:263)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:183)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:96)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:128)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) 
~[apache-cassandra-3.1.1.jar:3.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_65]
at 
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
 ~[apache-cassandra-3.1.1.jar:3.1.1]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
[apache-cassandra-3.1.1.jar:3.1.1]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

4 nodes out of 5 crashed with this error message. Now when I want to restart 
the first node I have the following error;

ERROR [main] 2016-01-14 17:15:59,617 JVMStabilityInspector.java:81 - Exiting 
due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation7465380878750576105dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.  Exception 
follows: org.apache.cassandra.serializers.MarshalException: Not enough bytes to 
read a map
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633)
 [apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556)
 [apache-cassandra-3.1.1.jar:3.1.1]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509)
 [apache-cassandra-3.1.1.jar:3.1.1]
at 

Re: Cassandra Java Driver

[Jean Tremblay]

Thank you very much Alexandre. That is very helpful.

Merry Christmas!!!

On 25 Dec 2015, at 16:38, Alexandre Dutra 
<alexandre.du...@datastax.com<mailto:alexandre.du...@datastax.com>> wrote:

Hi Jean,

You should use 3.0.0-beta1.

TL;DR

DataStax Java driver series 2.2.x has been discontinued in favor of series 3.x; 
we explained why in this mail to the Java driver mailing 
list<https://groups.google.com/a/lists.datastax.com/forum/m/#!topic/java-driver-user/Vu0cD0BguMc>.
 We do not advise users to use this series.

So the most recent driver version compatible with all versions of Cassandra, 
including 2.2 and 3.x, is now 3.0.0-beta1, although 3.0.0-rc1 will be released 
very soon.

In spite of its "beta" label, version 3.0.0-beta1 has been thoroughly tested 
against all versions of Cassandra and is definitely production-ready... as long 
as the Cassandra version in use is also production-ready. Note however that 
Cassandra 2.2 and 3.0 are quite recent and most companies AFAICT do not 
consider them yet as production-ready.

Hope that helps,

Alexandre


On Tue, Dec 22, 2015 at 4:40 PM Jean Tremblay 
<jean.tremb...@zen-innovations.com<mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Hi,
Which Java Driver is suited for Cassandra 2.2.x. ?
I see datastax 3.0.0 beta1 and datastax 2.2.0 rc3...
Are they suited for production?
Is there anything better?
Thanks for your comments and replies?
Jean
--
Alexandre Dutra
Driver & Tools Engineer @ DataStax

Cassandra Java Driver

[Jean Tremblay]

Hi,
Which Java Driver is suited for Cassandra 2.2.x. ?
I see datastax 3.0.0 beta1 and datastax 2.2.0 rc3...
Are they suited for production?
Is there anything better?
Thanks for your comments and replies?
Jean

Re: Written data is lost and no exception thrown back to the client

[Jean Tremblay]

I have the same problem.

When I bulk load my data, I have a problem with Cassandra Datastax driver.

dependency
groupIdcom.datastax.cassandra/groupId
artifactIdcassandra-driver-core/artifactId
version2.1.4/version !-- Driver 2.1.6, 2.1.7.1 gives problems. Some data 
is lost. --
/dependency

With version 2.1.6 and also with version 2.1.7.1 I have lost records with no 
error message what so ever.
With version 2.1.4 I have no missing records.

I use CL.ONE to write my records. I use RF 3.






On 21 Aug 2015, at 13:06 , Robert Wille 
rwi...@fold3.commailto:rwi...@fold3.com wrote:

But it shouldn’t matter. I have missing data, and no errors, which shouldn’t be 
possible except with CL=ANY.

FWIW, I’m working on some sample code so I can post a Jira.

Robert

On Aug 21, 2015, at 5:04 AM, Robert Wille 
rwi...@fold3.commailto:rwi...@fold3.com wrote:

RF=1 with QUORUM consistency. I know QUORUM is weird with RF=1, but it should 
be the same as ONE. If’s QUORUM instead of ONE because production has RF=3, and 
I was running this against my test cluster with RF=1.

On Aug 20, 2015, at 7:28 PM, Jason 
jkushm...@rocketfuelinc.commailto:jkushm...@rocketfuelinc.com wrote:

What consistency level were the writes?

From: Robert Willemailto:rwi...@fold3.com
Sent: ‎8/‎20/‎2015 18:25
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Written data is lost and no exception thrown back to the client

I wrote a data migration application which I was testing, and I pushed it too 
hard and the FlushWriter thread pool blocked, and I ended up with dropped 
mutation messages. I compared the source data against what is in my cluster, 
and as expected I have missing records. The strange thing is that my 
application didn’t error out. I’ve been doing some forensics, and there’s a lot 
about this that makes no sense and makes me feel very uneasy.

I use a lot of asynchronous queries, and I thought it was possible that I had 
bad error handling, so I checked for errors in other, independent ways.

I have a retry policy that on the first failure logs the error and then 
requests a retry. On the second failure it logs the error and then rethrows. A 
few retryable errors appeared in my logs, but no fatal errors. In theory, I 
should have a fatal error in my logs for any error that gets reported back to 
the client.

I wrap my Session object, and all queries go through this wrapper. This wrapper 
logs all query errors. Synchronous queries are wrapped in a try/catch which 
logs and rethrows. Asynchronous queries use a FutureCallback to log any 
onFailure invocations.

My logs indicate that no errors whatsoever were reported back to me. I do not 
understand how I can get dropped mutation messages and not know about it. I am 
running 2.0.16 with datastax Java driver 2.0.8. Three node cluster with RF=1. 
If someone could help me understand how this can occur, I would greatly 
appreciate it. A database that errors out is one thing. A database that errors 
out and makes you think everything was fine is quite another.

Thanks

Robert

Re: Nodetool repair with Load times 5

[Jean Tremblay]

, one with only three nodes. Configured 
exactly the same way, except that it has other seeds and another cluster name. 
I have done basically exactly the same manipulations, as I listed above.

On that second cluster, before I did the nodetool repair I had loads given 
from the “nodetool status” being around 350 GB for each node.

After the “nodetool repair” it sprang up to

nodetool status XYZdata
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  Owns (effective)  Host ID 
  Rack
UN  192.168.2.200  2.16 TB256 100.0%
1c28329b-f247-4e37-9664-85ee4492c46b  rack1
UN  192.168.2.201  4.09 TB256 100.0%
5afdb2cf-a57c-4b58-8f1f-65287b11dc3b  rack1
UN  192.168.2.202  4.68 TB256 100.0%
8e296000-809f-4fbc-a508-fcf7020556bd  rack1

——

Beside these very strange results on both clusters the systems seem to behave 
properly, as far as the select statements executed. The performance seems 
slightly reduced.

Mark Greene asked me if I did a nodetool repair. I actually tried to see it if 
it would change anything. Trying it returned instantly, which is odd.

Looking at the log files I see this:

nodetool -h nodet0 cleanup
INFO  [RMI TCP Connection(1538)-192.168.2.200] 2015-08-19 09:08:33,897 
CompactionManager.java:264 - No sstables for system_traces.sessions
INFO  [RMI TCP Connection(1538)-192.168.2.200] 2015-08-19 09:08:33,898 
CompactionManager.java:264 - No sstables for system_traces.events



—

Looking at the log files since I did the repair… I observe no errors not 
warnings except two type of warnings…

1) Warnings related to tombstones… which I think is normal.

WARN  [SharedPool-Worker-1] 2015-08-19 09:02:01,696 SliceQueryFilter.java:319 - 
Read 1329 live and 1097 tombstone cells in 
XYZdata.cohttp://XYZdata.co_rep_pcode for key: D:01 (see 
tombstone_warn_threshold). 5000 columns were requested, 
slices=[9:201001-9:201412:!]

2) Warnings related to partition size, which is probably related to my problem.

WARN  [CompactionExecutor:1600] 2015-08-19 03:18:00,869 SSTableWriter.java:240 
- Compacting large partition XYZdata/co_rep_pcode:D:820231 (117964258 bytes)

———


This has a very bad smell even though my system **seems** to work well. I’m 
quite lost and I will most probably reload all my data in both clusters.

Again, Alain, thanks for your help.

Kind regards

Jean






Anyway, see if you can give us more info related to this.

C*heers,

Alain



2015-08-18 14:40 GMT+02:00 Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com:
No. I did not try.
I would like to understand what is going on before I make my problem, maybe 
even worse.

I really would like to understand:

1) Is this normal?
2) What is the meaning of the column Load?
3) Is there anything to fix? Can I leave it like that?
  4) Did I do something wrong? When you use -par you only need to run 
repair from one node right? E.g.  nodetool -h 192.168.2.100 repair -pr -par -inc

Thanks for your feedback.

Jean

On 18 Aug 2015, at 14:33 , Mark Greene 
green...@gmail.commailto:green...@gmail.com wrote:

Hey Jean,

Did you try running a nodetool cleanup on all your nodes, perhaps one at a time?

On Tue, Aug 18, 2015 at 3:59 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I have a phenomena I cannot explain, and I would like to understand.

I’m running Cassandra 2.1.8 on a cluster of 5 nodes.
I’m using replication factor 3, with most default settings.

Last week I done a nodetool status which gave me on each node a load of about 
200 GB.
Since then there was no deletes no inserts.

This weekend I did a nodetool -h 192.168.2.100 repair -pr -par -inc

And now when I make a nodetool status I see completely a new picture!!

nodetool -h zennode0 status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  192.168.2.104  940.73 GB  256 ?   
c13e0858-091c-47c4-8773-6d6262723435  rack1
UN  192.168.2.100  1.07 TB256 ?   
c32a9357-e37e-452e-8eb1-57d86314b419  rack1
UN  192.168.2.101  189.03 GB  256 ?   
9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
UN  192.168.2.102  951.28 GB  256 ?   
8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
UN  192.168.2.103  196.54 GB  256 ?   
9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1

The nodes 192.168.2.101 and 103 are about what they were last week, but now the 
three other nodes have a load which is about 5 times bigger!

1) Is this normal?
2) What is the meaning of the column Load?
3) Is there anything to fix? Can I leave it like that?

Strange I’m asking to fix after I did a *repair*.

Thanks a lot for your help.

Kind regards

Jean

Nodetool repair with Load times 5

[Jean Tremblay]

Hi,

I have a phenomena I cannot explain, and I would like to understand.

I’m running Cassandra 2.1.8 on a cluster of 5 nodes.
I’m using replication factor 3, with most default settings.

Last week I done a nodetool status which gave me on each node a load of about 
200 GB.
Since then there was no deletes no inserts.

This weekend I did a nodetool -h 192.168.2.100 repair -pr -par -inc

And now when I make a nodetool status I see completely a new picture!!

nodetool -h zennode0 status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  192.168.2.104  940.73 GB  256 ?   
c13e0858-091c-47c4-8773-6d6262723435  rack1
UN  192.168.2.100  1.07 TB256 ?   
c32a9357-e37e-452e-8eb1-57d86314b419  rack1
UN  192.168.2.101  189.03 GB  256 ?   
9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
UN  192.168.2.102  951.28 GB  256 ?   
8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
UN  192.168.2.103  196.54 GB  256 ?   
9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1

The nodes 192.168.2.101 and 103 are about what they were last week, but now the 
three other nodes have a load which is about 5 times bigger!

1) Is this normal?
2) What is the meaning of the column Load?
3) Is there anything to fix? Can I leave it like that?

Strange I’m asking to fix after I did a *repair*.

Thanks a lot for your help.

Kind regards

Jean

Re: Nodetool repair with Load times 5

[Jean Tremblay]

No. I did not try.
I would like to understand what is going on before I make my problem, maybe 
even worse.

I really would like to understand:

1) Is this normal?
2) What is the meaning of the column Load?
3) Is there anything to fix? Can I leave it like that?
  4) Did I do something wrong? When you use -par you only need to run 
repair from one node right? E.g.  nodetool -h 192.168.2.100 repair -pr -par -inc

Thanks for your feedback.

Jean

On 18 Aug 2015, at 14:33 , Mark Greene 
green...@gmail.commailto:green...@gmail.com wrote:

Hey Jean,

Did you try running a nodetool cleanup on all your nodes, perhaps one at a time?

On Tue, Aug 18, 2015 at 3:59 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I have a phenomena I cannot explain, and I would like to understand.

I’m running Cassandra 2.1.8 on a cluster of 5 nodes.
I’m using replication factor 3, with most default settings.

Last week I done a nodetool status which gave me on each node a load of about 
200 GB.
Since then there was no deletes no inserts.

This weekend I did a nodetool -h 192.168.2.100 repair -pr -par -inc

And now when I make a nodetool status I see completely a new picture!!

nodetool -h zennode0 status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  192.168.2.104  940.73 GB  256 ?   
c13e0858-091c-47c4-8773-6d6262723435  rack1
UN  192.168.2.100  1.07 TB256 ?   
c32a9357-e37e-452e-8eb1-57d86314b419  rack1
UN  192.168.2.101  189.03 GB  256 ?   
9af90dea-90b3-4a8a-b88a-0aeabe3cea79  rack1
UN  192.168.2.102  951.28 GB  256 ?   
8eb7a5bb-6903-4ae1-a372-5436d0cc170c  rack1
UN  192.168.2.103  196.54 GB  256 ?   
9efc6f13-2b02-4400-8cde-ae831feb86e9  rack1

The nodes 192.168.2.101 and 103 are about what they were last week, but now the 
three other nodes have a load which is about 5 times bigger!

1) Is this normal?
2) What is the meaning of the column Load?
3) Is there anything to fix? Can I leave it like that?

Strange I’m asking to fix after I did a *repair*.

Thanks a lot for your help.

Kind regards

Jean

Re: Is there a way to remove a node with Opscenter?

[Jean Tremblay]

When you do a nodetool command and you don’t specify a hostname, it sends the 
requests via JMX to the localhost node. If that node is down then the command 
will not succeed.
In your case you are probably doing the command from a machine which has not 
cassandra running, in that case you need to specify a node with the switch -h.

So for your that would be:

nodetool -h a-node-ip-address removenode Host ID

where a-node-ip-address is the address of a server which has cassandra daemon 
running.

Cheers

Jean

On 08 Jul 2015, at 01:39 , Sid Tantia 
sid.tan...@baseboxsoftware.commailto:sid.tan...@baseboxsoftware.com wrote:

I tried both `nodetool remove node Host ID` and `nodetool decommission` and 
they both give the error:

nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection 
refused’.

Here is what I have tried to fix this:

1) Uncommented JVM_OPTS=”$JVM_OPTS -Djava.rmi.server.hostname=public name”
2) Changed rpc_address to 0.0.0.0
3) Restarted cassandra
4) Restarted datastax-agent

(Note that I installed my cluster using opscenter so that may have something to 
do with it? )




On Tue, Jul 7, 2015 at 2:08 PM, Surbhi Gupta 
surbhi.gupt...@gmail.commailto:surbhi.gupt...@gmail.com wrote:

If node is down use :

nodetool removenode Host ID

We have to run the below command when the node is down  if the cluster does 
not use vnodes, before running the nodetool removenode command, adjust the 
tokens.

If the node is up, then the command would be “nodetool decommission” to remove 
the node.


Remove the node from the “seed list” within the configuration cassandra.yaml.

On 7 July 2015 at 12:56, Sid Tantia 
sid.tan...@baseboxsoftware.commailto:sid.tan...@baseboxsoftware.com wrote:
Thanks for the response. I’m trying to remove a node that’s already down for 
some reason so its not allowing me to decommission it, is there some other way 
to do this?




On Tue, Jul 7, 2015 at 12:45 PM, Kiran mk 
coolkiran2...@gmail.commailto:coolkiran2...@gmail.com wrote:

Yes, if your intension is to decommission a node.  You can do that by clicking 
on the node and decommission.

Best Regards,
Kiran.M.K.

On Jul 8, 2015 1:00 AM, Sid Tantia 
sid.tan...@baseboxsoftware.commailto:sid.tan...@baseboxsoftware.com wrote:
I know you can use `nodetool removenode` from the command line but is there a 
way to remove a node from a cluster using OpsCenter?

Re: Restore Snapshots

[Jean Tremblay]

Good morning,
Alain, thank you so much. This is exactly what I needed.

 In my test I had a node which had for whatever reason the directory containing 
my data corrupted. I keep in a separate folder my snapshots.

Here are the steps I took to recover my sick node:

0) Cassandra is stopped on my sick node.
1) I wiped out my data directory. My snapshots were kept outside this directory.
2) I modified my Cassandra.yaml. I added auto_bootstrap: false .This is to make 
sure that my node does not synch with the others.
3) I restarted Cassandra. This step created a basic structure for my new data 
directory.
4) I did the command: nodetool resetlocalschema. This recreated all the folders 
for my cf.
5) I stopped Cassandra on my node.
6) I copied my snapshot in the right location. I actually hard linked them, 
this is very fast.
7) I restarted Cassandra.

That's it.

Thank you SO MUCH ALAIN for your support. You really helped me a lot.
On 25 Jun,2015, at 18:37, Alain RODRIGUEZ 
arodr...@gmail.commailto:arodr...@gmail.com wrote:

Hi Jean,

Answers in line to be sure to be exhaustive:

- how can I restore the data directory structure in order to copy my snapshots 
at the right position?
-- making a script to do it and testing it I would say. basically under any 
table repo you have a snapshots/snapshot_name directory (snapshot_name is 
timestamp if not specified off the top of my head..) and then your sstables.

- is it possible to recreate the schema on one node?
-- The easiest way that come to my mind is to set auto_bootstrap: false on a 
node not already in the ring. If you have trouble with the schema of a node in 
the ring run a nodetool resetlocalschema

- how can I avoid the node from streaming from the other nodes?
-- See above (auto_bootstrap: false). BTW, option might not be present at all, 
just add it.

- must I also have the snapshot of the system tables in order to restore a node 
from only the snapshot of my tables?
-- just you user table. Yet remember that snapshot is per node and as such you 
will just have part of the data this node use to hold. meaning that if the new 
node have different tokens, there will be unused data + missing data for sure.

Basically when a node is down I use to remove it, repair the cluster, and 
bootstap it (auto_bootstrap: true). Streams are part of Cassandra. I accept 
that. An other solution would be to replace the node -- 
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html

C*heers,

Alain

2015-06-25 17:07 GMT+02:00 Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com:
Hi,

I am testing snapshot restore procedures in case of a major catastrophe on our 
cluster. I'm using Cassandra 2.1.7 with RF:3

The scenario that I am trying to solve is how to quickly get one node back to 
work after its disk failed and lost all its data assuming that the only thing I 
have is its snapshots.

The procedure that I'm following is the one explained here: 
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html

I can do a snapshot that is straight forward.
My problem is in the restore of the snapshot.

If I restart Cassandra with an empty data directory the node will bootstrap.
Bootstrap is very nice, since it recreate the schema and reload the data from 
its neighbour.
But this is quite heavy traffic and quite a slow process.

My questions are:

- how can I restore the data directory structure in order to copy my snapshots 
at the right position?
- is it possible to recreate the schema on one node?
- how can I avoid the node from streaming from the other nodes?
- must I also have the snapshot of the system tables in order to restore a node 
from only the snapshot of my tables?

Thanks for your comments.

Jean

Restore Snapshots

[Jean Tremblay]

Hi,

I am testing snapshot restore procedures in case of a major catastrophe on our 
cluster. I’m using Cassandra 2.1.7 with RF:3

The scenario that I am trying to solve is how to quickly get one node back to 
work after its disk failed and lost all its data assuming that the only thing I 
have is its snapshots.

The procedure that I’m following is the one explained here: 
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html

I can do a snapshot that is straight forward.
My problem is in the restore of the snapshot.

If I restart Cassandra with an empty data directory the node will bootstrap.
Bootstrap is very nice, since it recreate the schema and reload the data from 
its neighbour.
But this is quite heavy traffic and quite a slow process.

My questions are:

- how can I restore the data directory structure in order to copy my snapshots 
at the right position?
- is it possible to recreate the schema on one node?
- how can I avoid the node from streaming from the other nodes?
- must I also have the snapshot of the system tables in order to restore a node 
from only the snapshot of my tables?

Thanks for your comments.

Jean

Re: After Restart Nodes had lost data

[Jean Tremblay]

No, I did not.



On 24 Jun 2015, at 06:05, Jason Wee 
peich...@gmail.commailto:peich...@gmail.com wrote:

on the node 192.168.2.100, did you run repair after its status is UN?

On Wed, Jun 24, 2015 at 2:46 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Dear Alain,

Thank you for your reply.

Ok, yes I did not drain. The cluster was loaded with tons of records, and no 
new records were added since few weeks. Each node had a load of about 160 GB as 
seen in the “nodetool status. I killed the cassandradeamon, and restarted it. 
After cassandra was restarted, I could see in the “nodetool status” a load of 5 
GB!!

I don’t use counters.
I use RF 3 on 5 nodes. I did not change the replication factor.
I have two types of read queries. One use QUORUM for read, and the other use 
ONE for consistency level.
I did not change the Topology.

Are you sure this node had data before you restart it ?

Actually the full story is:

- I stopped node0(192.168.2.100), and I restarted it.
- I stopped node1(192.168.2.101).
- I made a nodetool status and I noticed that node0 was UN and had a load 5 GB. 
I found this really weird because all the other ones had about 160GB. I also 
saw that node1 was DN with a load of about 160GB.
- I restarted node1.
- I made a nodetool status and I noticed that node1 was UN and had a load of 
also 5GB, it previously had a load of about 160GB. That I’m sure.
- Then my program could no longer query C*. Neither the QUORUM nor the ONE 
consistency level statements could read data.

What does a nodetool status mykeyspace outputs ?

I cannot try this anymore. I flushed the whole cluster, and I am currently 
reloading everything. I was too much in a hurry. I have a demo tomorrow, and I 
will manage to have it back before tomorrow.

After my bad decision of flushing the cluster, I realised that I could have 
bootstrapped again my two nodes. Learning by doing.

It’s like the whole cluster is paralysed -- what does it mean, be more 
accurate on this please.

You should tell us action that were taken before this occurred and now what is 
not working since a C* cluster in this state could perfectly run. No SPOF.

What I did before? Well this cluster was basically idling. I was only making 
lots of select on it. I was loaded since few weeks.
But what I noticed when I restarted node0 is the following:

INFO  [InternalResponseStage:1] 2015-06-23 11:45:32,723 
ColumnFamilyStore.java:882 - Enqueuing flush of schema_columnfamilies: 131587 
(0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:2] 2015-06-23 11:45:32,723 Memtable.java:346 - 
Writing Memtable-schema_columnfamilies@917967643(34850 serialized bytes, 585 
ops, 0%/0% of on/off-heap limit)
 WARN  [GossipTasks:1] 2015-06-23 11:45:33,459 FailureDetector.java:251 - 
 Not marking nodes down due to local pause of 25509152054  50
INFO  [MemtableFlushWriter:1] 2015-06-23 11:45:33,982 Memtable.java:385 - 
Completed flushing 
/home/maia/apache-cassandra-DATA/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-11-Data.db
 (5274 bytes) for commitlog position ReplayPos
ition(segmentId=1435052707645, position=144120)
INFO  [GossipStage:1] 2015-06-23 11:45:33,985 StorageService.java:1642 - Node 
/192.168.2.101http://192.168.2.101 state jump to normal
INFO  [GossipStage:1] 2015-06-23 11:45:33,991 Gossiper.java:987 - Node 
/192.168.2.102http://192.168.2.102 has restarted, now UP
INFO  [SharedPool-Worker-1] 2015-06-23 11:45:33,992 Gossiper.java:954 - 
InetAddress /192.168.2.102http://192.168.2.102 is now UP
INFO  [HANDSHAKE-/192.168.2.102http://192.168.2.102] 2015-06-23 11:45:33,993 
OutboundTcpConnection.java:485 - Handshaking version with 
/192.168.2.102http://192.168.2.102
INFO  [GossipStage:1] 2015-06-23 11:45:33,993 StorageService.java:1642 - Node 
/192.168.2.102http://192.168.2.102 state jump to normal
INFO  [GossipStage:1] 2015-06-23 11:45:33,999 Gossiper.java:987 - Node 
/192.168.2.103http://192.168.2.103 has restarted, now UP
INFO  [SharedPool-Worker-1] 2015-06-23 11:45:33,999 Gossiper.java:954 - 
InetAddress /192.168.2.103http://192.168.2.103 is now UP
INFO  [GossipStage:1] 2015-06-23 11:45:34,001 StorageService.java:1642 - Node 
/192.168.2.103http://192.168.2.103 state jump to normal
INFO  [HANDSHAKE-/192.168.2.103http://192.168.2.103] 2015-06-23 11:45:34,020 
OutboundTcpConnection.java:485 - Handshaking version with 
/192.168.2.103http://192.168.2.103
INFO  [main] 2015-06-23 11:45:34,021 StorageService.java:1642 - Node 
zennode0/192.168.2.100http://192.168.2.100 state jump to normal
INFO  [GossipStage:1] 2015-06-23 11:45:34,028 StorageService.java:1642 - Node 
/192.168.2.104http://192.168.2.104 state jump to normal
INFO  [main] 2015-06-23 11:45:34,038 CassandraDaemon.java:583 - Waiting for 
gossip to settle before accepting client requests...
INFO  [GossipStage:1] 2015-06-23 11:45:34,039 StorageService.java:1642 - Node 
/192.168.2.101http://192.168.2.101 state jump to normal
INFO

Re: After Restart Nodes had lost data

[Jean Tremblay]

Does anyone know what to do when such an event occurs?
Does anyone know how this could happen?

I would have tried repairing the node with nodetool repair but that takes much 
too long… I need my cluster to work for our online system. At this point 
nothing is working. It’s like the whole cluster is paralysed.
The only solution I see is to remove temporarily that node.

Thanks for your comments.

On 23 Jun 2015, at 12:45 , Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:

Hi,

I have a cluster with 5 nodes running Cassandra 2.1.6.

I had to reboot a node. I killed the cassandra process on that node. Rebooted 
the machine, and restarted Cassandra.

~/apache-cassandra-DATA/data:321nodetool status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  192.168.2.104  158.27 GB  256 ?   
6479205e-6a19-49a8-b1a1-7e788ec29caa  rack1
UN  192.168.2.100  4.75 GB256 ?   
e821da50-23c6-4ea0-b3a1-275ded63bc1f  rack1
UN  192.168.2.101  157.43 GB  256 ?   
01525665-bacc-4207-a8c3-eb4fd9532401  rack1
UN  192.168.2.102  159.27 GB  256 ?   
596a33d7-5089-4c7e-a9ad-e1f22111b160  rack1
UN  192.168.2.103  167 GB 256 ?   
0ce1d48e-57a9-4615-8e12-d7ef3d621c7d  rack1


After restarting node 192.168.2.100 I noticed that its load was diminish to 2%. 
And now when I query the cluster my queries are bombing and that node times out 
with an error

WARN  [MessagingService-Incoming-/192.168.2.102] 2015-06-23 12:26:00,056 
IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from 
socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=ddc346b0-1372-11e5-9ba1-195596ed1fd9
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
 ~[apache-cassandra-2.1.6.jar:2.1.6]

What is going on? Did anyone live something like that?

Re: Datastax Java Driver vs Cassandra 2.1.7

[Jean Tremblay]

I agree. Thanks a lot.
On 23 Jun 2015, at 15:31 , Sam Tunnicliffe 
s...@beobal.commailto:s...@beobal.com wrote:

Although amending the query is a workaround for this (and duplicating the 
columns in the selection is not something I imagine one would deliberately do), 
this is still an ugly regression, so I've opened 
https://issues.apache.org/jira/browse/CASSANDRA-9636 to fix it.

Thanks,
Sam

On Tue, Jun 23, 2015 at 1:52 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi Sam,

You have a real good gut feeling.
I went to see the query that I used since many months… which was working…. but 
obviously there is something wrong with it.
The problem with it was *simply* that I placed twice the same field in the 
select. I corrected in my code and now I don’t have the error with 2.1.7.

This provocated the error on the nodes:
ERROR [SharedPool-Worker-1] 2015-06-23 10:56:01,186 Message.java:538 - 
Unexpected exception during request; channel = [id: 0x5e809aa1, 
/192.168.2.8:49581http://192.168.2.8:49581/ = 
/192.168.2.201:9042http://192.168.2.201:9042/]
java.lang.AssertionError: null
at org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) 
~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:347)
 ~[apache-cassandra-2.1.7.jar:2.1.7]

I can also reproduce the error on cqlsh:

cqlsh select c1, p1, mm, c2, iq, iq from ds.t1 where type='D' and c1=1 and 
mm=201401 and mm=201402 and p1='01';
ServerError: ErrorMessage code= [Server error] 
message=java.lang.AssertionError
cqlsh select c1, p1, mm, c2, iq  from ds.t1 where type='D' and c1=1 and 
mm=201401 and mm=201402 and p1='01';

 c1 | p1| mm | c2 | iq
+---+++-
  1 |01 | 201401 |  1 |   {‘XX': 97160}
…

Conclusion… my mistake. Sorry.


On 23 Jun 2015, at 13:06 , Sam Tunnicliffe 
s...@beobal.commailto:s...@beobal.com wrote:

Can you share the query that you're executing when you see the error and the 
schema of the target table? It could be something related to CASSANDRA-9532.

On Tue, Jun 23, 2015 at 10:05 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I’m using Datastax Java Driver V 2.1.6
I migrated my cluster to Cassandra V2.1.7
And now I have an error on my client that goes like:

2015-06-23 10:49:11.914  WARN 20955 --- [ I/O worker #14] 
com.datastax.driver.core.RequestHandler  : 
/192.168.2.201:9042http://192.168.2.201:9042/ replied with server error 
(java.lang.AssertionError), trying next host.

And on the node I have an NPE

ERROR [SharedPool-Worker-1] 2015-06-23 10:56:01,186 Message.java:538 - 
Unexpected exception during request; channel = [id: 0x5e809aa1, 
/192.168.2.8:49581http://192.168.2.8:49581/ = 
/192.168.2.201:9042http://192.168.2.201:9042/]
java.lang.AssertionError: null
at org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) 
~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:347)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.processColumnFamily(SelectStatement.java:1289)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1223)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:299)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:238)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:67)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:238)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:493)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:134)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
 [apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
 [apache-cassandra-2.1.7.jar:2.1.7]
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext$8.run

Re: Datastax Java Driver vs Cassandra 2.1.7

[Jean Tremblay]

Hi Sam,

You have a real good gut feeling.
I went to see the query that I used since many months… which was working…. but 
obviously there is something wrong with it.
The problem with it was *simply* that I placed twice the same field in the 
select. I corrected in my code and now I don’t have the error with 2.1.7.

This provocated the error on the nodes:
ERROR [SharedPool-Worker-1] 2015-06-23 10:56:01,186 Message.java:538 - 
Unexpected exception during request; channel = [id: 0x5e809aa1, 
/192.168.2.8:49581http://192.168.2.8:49581/ = 
/192.168.2.201:9042http://192.168.2.201:9042/]
java.lang.AssertionError: null
at org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) 
~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:347)
 ~[apache-cassandra-2.1.7.jar:2.1.7]

I can also reproduce the error on cqlsh:

cqlsh select c1, p1, mm, c2, iq, iq from ds.t1 where type='D' and c1=1 and 
mm=201401 and mm=201402 and p1='01';
ServerError: ErrorMessage code= [Server error] 
message=java.lang.AssertionError
cqlsh select c1, p1, mm, c2, iq  from ds.t1 where type='D' and c1=1 and 
mm=201401 and mm=201402 and p1='01';

 c1 | p1| mm | c2 | iq
+---+++-
  1 |01 | 201401 |  1 |   {‘XX': 97160}
…

Conclusion… my mistake. Sorry.


On 23 Jun 2015, at 13:06 , Sam Tunnicliffe 
s...@beobal.commailto:s...@beobal.com wrote:

Can you share the query that you're executing when you see the error and the 
schema of the target table? It could be something related to CASSANDRA-9532.

On Tue, Jun 23, 2015 at 10:05 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I’m using Datastax Java Driver V 2.1.6
I migrated my cluster to Cassandra V2.1.7
And now I have an error on my client that goes like:

2015-06-23 10:49:11.914  WARN 20955 --- [ I/O worker #14] 
com.datastax.driver.core.RequestHandler  : 
/192.168.2.201:9042http://192.168.2.201:9042/ replied with server error 
(java.lang.AssertionError), trying next host.

And on the node I have an NPE

ERROR [SharedPool-Worker-1] 2015-06-23 10:56:01,186 Message.java:538 - 
Unexpected exception during request; channel = [id: 0x5e809aa1, 
/192.168.2.8:49581http://192.168.2.8:49581/ = 
/192.168.2.201:9042http://192.168.2.201:9042/]
java.lang.AssertionError: null
at org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) 
~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:347)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.processColumnFamily(SelectStatement.java:1289)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1223)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:299)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:238)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:67)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:238)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:493)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:134)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
 [apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
 [apache-cassandra-2.1.7.jar:2.1.7]
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_45]
at 
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
 [apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
[apache-cassandra-2.1.7.jar:2.1.7]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

Is there a known problem on Cassandra 2.1.7

Re: After Restart Nodes had lost data

[Jean Tremblay]

.jar:2.1.6]
at org.apache.cassandra.db.Row$RowSerializer.deserialize(Row.java:74) 
~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:97)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:69)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) 
~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:188)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:170)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
WARN  [MessagingService-Incoming-/192.168.2.102] 2015-06-23 11:46:00,629 
IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from 
socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=ddc346b0-1372-11e5-9ba1-195596ed1fd9
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:89)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.db.Row$RowSerializer.deserialize(Row.java:74) 
~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:97)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:69)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) 
~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:188)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:170)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
WARN  [MessagingService-Incoming-/192.168.2.103] 2015-06-23 11:46:00,633 
IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from 
socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=ddc346b0-1372-11e5-9ba1-195596ed1fd9
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:89)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.db.Row$RowSerializer.deserialize(Row.java:74) 
~[apache-cassandra-2.1.6.jar:2.1.6]


That seems to me more nasty. I don’t know what it means, but I have the 
impression that the nodes which I did not touch seem now to talk with node0 
about a table which the latter does not know about!!! A table node0 “forgot”.


Well I don’t ask for miracle here. If I would have noticed that there was 
already a problem after restarted the first node I would have simply fixed that 
node before doing anything else… but I did not noticed this.

Thanks for your help...




On 23 Jun 2015, at 19:24 , Alain RODRIGUEZ 
arodr...@gmail.commailto:arodr...@gmail.com wrote:

Hi Jean,

I had to reboot a node. I killed the cassandra process on that node. You 
should drain the node before killing java (or using any service stop command). 
This is not what causes the issue yet it will help you to keep consistence if 
you use counters, and make the reboot faster in any cases.

What is going on highly depends on what you did before.

Did you change the RF ?
Did you change the Topology ?
Are you sure this node had data before you restart it ?
What does a nodetool status mykeyspace outputs ?

To make the join faster you could try to bootstrap the node again. I just hope 
you have a RF  1 (btw, you have one replica down, if you want to still have 
Reads / Writes working, take care of having a Consistency Level low enough).

It’s like the whole cluster is paralysed -- what does it mean, be more 
accurate on this please.

You should tell us action that were taken before this occurred and now what is 
not working since a C* cluster in this state could perfectly run. No SPOF.

C*heers

2015-06-23 16:10 GMT+02:00 Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com:
Does anyone know what to do when such an event occurs?
Does anyone know how this could happen?

I would have tried repairing

Datastax Java Driver vs Cassandra 2.1.7

[Jean Tremblay]

Hi,

I’m using Datastax Java Driver V 2.1.6
I migrated my cluster to Cassandra V2.1.7
And now I have an error on my client that goes like:

2015-06-23 10:49:11.914  WARN 20955 --- [ I/O worker #14] 
com.datastax.driver.core.RequestHandler  : /192.168.2.201:9042 replied with 
server error (java.lang.AssertionError), trying next host.

And on the node I have an NPE

ERROR [SharedPool-Worker-1] 2015-06-23 10:56:01,186 Message.java:538 - 
Unexpected exception during request; channel = [id: 0x5e809aa1, 
/192.168.2.8:49581 = /192.168.2.201:9042]
java.lang.AssertionError: null
at org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) 
~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:347)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.processColumnFamily(SelectStatement.java:1289)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1223)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:299)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:238)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:67)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:238)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:493)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:134)
 ~[apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
 [apache-cassandra-2.1.7.jar:2.1.7]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
 [apache-cassandra-2.1.7.jar:2.1.7]
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at 
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
 [netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_45]
at 
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
 [apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
[apache-cassandra-2.1.7.jar:2.1.7]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

Is there a known problem on Cassandra 2.1.7?

Thanks for your comments.

Jean

After Restart Nodes had lost data

[Jean Tremblay]

Hi,

I have a cluster with 5 nodes running Cassandra 2.1.6.

I had to reboot a node. I killed the cassandra process on that node. Rebooted 
the machine, and restarted Cassandra.

~/apache-cassandra-DATA/data:321nodetool status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  192.168.2.104  158.27 GB  256 ?   
6479205e-6a19-49a8-b1a1-7e788ec29caa  rack1
UN  192.168.2.100  4.75 GB256 ?   
e821da50-23c6-4ea0-b3a1-275ded63bc1f  rack1
UN  192.168.2.101  157.43 GB  256 ?   
01525665-bacc-4207-a8c3-eb4fd9532401  rack1
UN  192.168.2.102  159.27 GB  256 ?   
596a33d7-5089-4c7e-a9ad-e1f22111b160  rack1
UN  192.168.2.103  167 GB 256 ?   
0ce1d48e-57a9-4615-8e12-d7ef3d621c7d  rack1


After restarting node 192.168.2.100 I noticed that its load was diminish to 2%. 
And now when I query the cluster my queries are bombing and that node times out 
with an error

WARN  [MessagingService-Incoming-/192.168.2.102] 2015-06-23 12:26:00,056 
IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from 
socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find 
cfId=ddc346b0-1372-11e5-9ba1-195596ed1fd9
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
 ~[apache-cassandra-2.1.6.jar:2.1.6]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
 ~[apache-cassandra-2.1.6.jar:2.1.6]

What is going on? Did anyone live something like that?

Detect Repair is finish

[Jean Tremblay]

Hi,

What is the best way to see if a repair is finished? Is there a JMX object or 
is there a command to see if a repair is finished?
What happens if by mistake an operator starts a repair before the previous is 
not yet finished? Will they execute both one after the other or at the same 
time?

Thanks for your help.

Jean

Re: nodetool repair

[Jean Tremblay]

Perfect thank you.
So making a weekly nodetool repair -pr”  on all nodes one after the other will 
repair my cluster. That is great.

If it does a compaction, does it mean that it would also clean up my tombstone 
from my LeveledCompactionStrategy tables at the same time?

Thanks for your help.

On 19 Jun 2015, at 07:56 , arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com wrote:

Hi Jean,

Running nodetool repair on a node will repair only that node in the cluster. It 
is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range 
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.




--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Long nodetool repair use nodetool repair incremental

[Jean Tremblay]

Hi,

I understand that we must repair the DB on a regular basis.
Now I also see that making a repair is using lots of resources in the cluster 
so I need to do this during the weekend because I really would like to have 
high performance at least during the week days.

In the documentation I see the incremental nodetool repair”. That really seems 
to be the option for me.

Few questions:

1) Why is it written in the documentation that it is suggested to do 
incremental repair daily, and full repair on the weekends? Is repair 
incremental not good enough or not safe?
2) In order to use incremental repair we need to do migration steps. One of 
these steps is to run the tool sstablerepairedset. What is the parameter 
sstables? Is it the name (without extension) of all *.db files contained in 
the data/cfdir?
3) Is the sstablerepairedset really needed when making a repair incremental on 
an empty DB?

Thanks for your help

Jean

Re: nodetool repair

[Jean Tremblay]

Thank you for your reply.



On 19 Jun 2015, at 20:36, 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com wrote:

It seems to me that running repair on any given node may also induce repairs to 
related replica nodes. For example, if I run repair on node A and node B has 
some replicas, data might stream from A to B (assuming A has newer/more data). 
Now, that does NOT mean that node B will be fully repaired. You still need to 
run repair -pr on all nodes before gc_grace_seconds.

You can run repairs on multiple nodes at the same time. However, you might end 
up with a large amount of streaming, if many repairs are needed. So, you should 
be aware of a performance impact.

I run weekly repairs on one node at a time, if possible. On, larger rings, 
though, I run repairs on multiple nodes staggered by a few hours. Once your 
routine maintenance is established, repairs will not run for very long. But, if 
you have a large ring that hasn’t been repaired, those first repairs may take 
days (but should get faster as you get further through the ring).


Sean Durity

From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Friday, June 19, 2015 3:56 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: nodetool repair

Hi,

This is not necessarily true. Repair will induce compactions only if you have 
entropy in your cluster. If not it will just read your data to compare all the 
replica of each piece of data (using indeed cpu and disk IO).

If there is some data missing it will repair it. Though, due to merkle tree 
size, you will generally stream more data than just the data needed. To limit 
this downside and the compactions amount, use range repairs -- 
http://www.datastax.com/dev/blog/advanced-repair-techniques.

About tombstones, they will be evicted only after gc_grace_period and only if 
all the parts of the row are part of the compaction.

C*heers,

Alain

2015-06-19 9:08 GMT+02:00 arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com:
Yes compactions will remove tombstones

On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Perfect thank you.
So making a weekly nodetool repair -pr”  on all nodes one after the other will 
repair my cluster. That is great.

If it does a compaction, does it mean that it would also clean up my tombstone 
from my LeveledCompactionStrategy tables at the same time?

Thanks for your help.

On 19 Jun 2015, at 07:56 , arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com wrote:

Hi Jean,

Running nodetool repair on a node will repair only that node in the cluster. It 
is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range 
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?
If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.




--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http

nodetool repair

[Jean Tremblay]

Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.

Re: nodetool repair

[Jean Tremblay]

Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.

Missing data

[Jean Tremblay]

Hi,

I have reloaded the data in my cluster of 3 nodes RF: 2.
I have loaded about 2 billion rows in one table.
I use LeveledCompactionStrategy on my table.
I use version 2.1.6.
I use the default cassandra.yaml, only the ip address for seeds and throughput 
has been change.

I loaded my data with simple insert statements. This took a bit more than one 
day to load the data… and one more day to compact the data on all nodes.
For me this is quite acceptable since I should not be doing this again.
I have done this with previous versions like 2.1.3 and others and I basically 
had absolutely no problems.

Now I read the log files on the client side, there I see no warning and no 
errors.
On the nodes side there I see many WARNING, all related with tombstones, but 
there are no ERRORS.

My problem is that I see some *many missing records* in the DB, and I have 
never observed this with previous versions.

1) Is this a know problem?
2) Do you have any idea how I could track down this problem?
3) What is the meaning of this WARNING (the only type of ERROR | WARN  I could 
find)?

WARN  [SharedPool-Worker-2] 2015-06-15 10:12:00,866 SliceQueryFilter.java:319 - 
Read 2990 live and 16016 tombstone cells in gttdata.alltrades_co_rep_pcode for 
key: D:07 (see tombstone_warn_threshold). 5000 columns were requested, 
slices=[388:201001-388:201412:!]


4) Is it possible to have Tombstone when we make no DELETE statements?

I’m lost…

Thanks for your help.

Catastrophy Recovery.

[Jean Tremblay]

Hi,

I have a cluster of 3 nodes RF: 2.
There are about 2 billion rows in one table.
I use LeveledCompactionStrategy on my table.
I use version 2.1.6.
I use the default cassandra.yaml, only the ip address for seeds and throughput 
has been change.

I am have tested a scenario where one node crashes and loose all its data.
I have deleted all data on this node after having stopped Cassandra.
At this point I noticed that the cluster was giving proper results. What I was 
expecting from a cluster DB.

I then restarted that node and I observed that the node was joining the cluster.
After an hour or so the old “defect” node was up and normal.
I noticed that its hard disk loaded with much less data than its neighbours.

When I was querying the DB, the cluster was giving me different results for 
successive identical queries.
I guess the old “defect” node was giving me less rows than it should have.

1) For what I understand, if you have a fixed node with no data it will 
automatically bootstrap and recover all its old data from its neighbour while 
doing the joining phase. Is this correct?
2) After such catastrophe, and after the joining phase is done should the 
cluster not be ready to deliver always consistent data if there was no inserts 
or delete during the catastrophe?
3) After the bootstrap of a broken node is finish, i.e. after the joining 
phase, is there not simply a repair to be done on that node using “node repair?


Thanks for your comments.

Kind regards

Jean

Re: Catastrophy Recovery.

[Jean Tremblay]

That is really wonderful. Thank you very much Alain. You gave me a lot of 
trails to investigate. Thanks again for you help.

On 15 Jun 2015, at 17:49 , Alain RODRIGUEZ 
arodr...@gmail.commailto:arodr...@gmail.com wrote:

Hi, it looks like your starting to use Cassandra.

Welcome.

I invite you to read from here as much as you can 
http://docs.datastax.com/en/cassandra/2.1/cassandra/gettingStartedCassandraIntro.html.

When a node lose some data you have various anti entropy mechanism

Hinted Handoff -- For writes that occurred while node was down and known as 
such by other nodes (exclusively)
Read repair -- On each read, you can set a chance to check other nodes for 
auto correction.
Repair ( called either manual / anti entropy / full / ...) : Which takes care 
to give back a node its missing data only for the range this node handles (-pr) 
or for all its data (its range plus its replica). This is something you 
generally want to perform on all nodes on a regular basis (lower than the 
lowest gc_grace_period set on any of your tables).

Also, you are having wrong values because you probably have a Consistency Level 
(CL) too low. If you want this to never happen you have to set Read (R) / Write 
(W) consistency level as follow : R + W  RF (Refplication Factor), if not you 
can see what you are currently seeing. I advise you to set your consistency to 
local_quorum or quorum on single DC environment. Also, with 3 nodes, you 
should set RF to 3, if not you won't be able to reach a strong consistency due 
to the formula I just give you.

There is a lot more to know, you should read about this all. Using Cassandra 
without knowing about its internals would lead you to very poor and unexpected 
results.

To answer your questions:

For what I understand, if you have a fixed node with no data it will 
automatically bootstrap and recover all its old data from its neighbour while 
doing the joining phase. Is this correct?

-- Not at all, unless it join the ring for the first time, which is not your 
case. Through it will (by default) slowly recover while you read.

After such catastrophe, and after the joining phase is done should the cluster 
not be ready to deliver always consistent data if there was no inserts or 
delete during the catastrophe?

No, we can't ensure that, excepted dropping the node and bootstrapping a new 
one. What we can make sure of is that there is enough replica remaining to 
serve consistent data (search for RF and CL)

After the bootstrap of a broken node is finish, i.e. after the joining phase, 
is there not simply a repair to be done on that node using “node repair?

This sentence is false bootstrap / joining phase ≠ from broken node coming 
back. You are right on repair, if a broken node (or down for too long - default 
3 hours) come back you have to repair. But repair is slow, make sure you can 
afford a node, see my previous answer.

Testing is a really good idea but you also have to read a lot imho.

Good luck,

C*heers,

Alain


2015-06-15 11:13 GMT+02:00 Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com:

Hi,

I have a cluster of 3 nodes RF: 2.
There are about 2 billion rows in one table.
I use LeveledCompactionStrategy on my table.
I use version 2.1.6.
I use the default cassandra.yaml, only the ip address for seeds and throughput 
has been change.

I am have tested a scenario where one node crashes and loose all its data.
I have deleted all data on this node after having stopped Cassandra.
At this point I noticed that the cluster was giving proper results. What I was 
expecting from a cluster DB.

I then restarted that node and I observed that the node was joining the cluster.
After an hour or so the old “defect” node was up and normal.
I noticed that its hard disk loaded with much less data than its neighbours.

When I was querying the DB, the cluster was giving me different results for 
successive identical queries.
I guess the old “defect” node was giving me less rows than it should have.

1) For what I understand, if you have a fixed node with no data it will 
automatically bootstrap and recover all its old data from its neighbour while 
doing the joining phase. Is this correct?
2) After such catastrophe, and after the joining phase is done should the 
cluster not be ready to deliver always consistent data if there was no inserts 
or delete during the catastrophe?
3) After the bootstrap of a broken node is finish, i.e. after the joining 
phase, is there not simply a repair to be done on that node using “node repair?


Thanks for your comments.

Kind regards

Jean

Re: Missing data

[Jean Tremblay]

Thanks Robert, but I don’t insert NULL values, but thanks anyway.

On 15 Jun 2015, at 19:16 , Robert Wille 
rwi...@fold3.commailto:rwi...@fold3.com wrote:

You can get tombstones from inserting null values. Not sure if that’s the 
problem, but it is another way of getting tombstones in your data.

On Jun 15, 2015, at 10:50 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:

Dear all,

I identified a bit more closely the root cause of my missing data.

The problem is occurring when I use

dependency
groupIdcom.datastax.cassandra/groupId
artifactIdcassandra-driver-core/artifactId
version2.1.6/version
/dependency

on my client against Cassandra 2.1.6.

I did not have the problem when I was using the driver 2.1.4 with C* 2.1.4.
Interestingly enough I don’t have the problem with the driver 2.1.4 with C* 
2.1.6.  !!

So as far as I can locate the problem, I would say that the version 2.1.6 of 
the driver is not working properly and is loosing some of my records.!!!

——

As far as my tombstones are concerned I don’t understand their origin.
I removed all location in my code where I delete items, and I do not use TTL 
anywhere ( I don’t need this feature in my project).

And yet I have many tombstones building up.

Is there another origin for tombstone beside TTL, and deleting items? Could the 
compaction of LeveledCompactionStrategy be the origin of them?

@Carlos thanks for your guidance.

Kind regards

Jean



On 15 Jun 2015, at 11:17 , Carlos Rolo 
r...@pythian.commailto:r...@pythian.com wrote:

Hi Jean,

The problem of that Warning is that you are reading too many tombstones per 
request.

If you do have Tombstones without doing DELETE it because you probably TTL'ed 
the data when inserting (By mistake? Or did you set default_time_to_live in 
your table?). You can use nodetool cfstats to see how many tombstones per read 
slice you have. This is, probably, also the cause of your missing data. Data 
was tombstoned, so it is not available.



Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolohttp://linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.comhttp://www.pythian.com/

On Mon, Jun 15, 2015 at 10:54 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I have reloaded the data in my cluster of 3 nodes RF: 2.
I have loaded about 2 billion rows in one table.
I use LeveledCompactionStrategy on my table.
I use version 2.1.6.
I use the default cassandra.yaml, only the ip address for seeds and throughput 
has been change.

I loaded my data with simple insert statements. This took a bit more than one 
day to load the data… and one more day to compact the data on all nodes.
For me this is quite acceptable since I should not be doing this again.
I have done this with previous versions like 2.1.3 and others and I basically 
had absolutely no problems.

Now I read the log files on the client side, there I see no warning and no 
errors.
On the nodes side there I see many WARNING, all related with tombstones, but 
there are no ERRORS.

My problem is that I see some *many missing records* in the DB, and I have 
never observed this with previous versions.

1) Is this a know problem?
2) Do you have any idea how I could track down this problem?
3) What is the meaning of this WARNING (the only type of ERROR | WARN  I could 
find)?

WARN  [SharedPool-Worker-2] 2015-06-15 10:12:00,866 SliceQueryFilter.java:319 - 
Read 2990 live and 16016 tombstone cells in gttdata.alltrades_co_rep_pcode for 
key: D:07 (see tombstone_warn_threshold). 5000 columns were requested, 
slices=[388:201001-388:201412:!]


4) Is it possible to have Tombstone when we make no DELETE statements?

I’m lost…

Thanks for your help.



--

Re: Missing data

[Jean Tremblay]

Dear all,

I identified a bit more closely the root cause of my missing data.

The problem is occurring when I use

dependency
groupIdcom.datastax.cassandra/groupId
artifactIdcassandra-driver-core/artifactId
version2.1.6/version
/dependency

on my client against Cassandra 2.1.6.

I did not have the problem when I was using the driver 2.1.4 with C* 2.1.4.
Interestingly enough I don’t have the problem with the driver 2.1.4 with C* 
2.1.6.  !!

So as far as I can locate the problem, I would say that the version 2.1.6 of 
the driver is not working properly and is loosing some of my records.!!!

——

As far as my tombstones are concerned I don’t understand their origin.
I removed all location in my code where I delete items, and I do not use TTL 
anywhere ( I don’t need this feature in my project).

And yet I have many tombstones building up.

Is there another origin for tombstone beside TTL, and deleting items? Could the 
compaction of LeveledCompactionStrategy be the origin of them?

@Carlos thanks for your guidance.

Kind regards

Jean



On 15 Jun 2015, at 11:17 , Carlos Rolo 
r...@pythian.commailto:r...@pythian.com wrote:

Hi Jean,

The problem of that Warning is that you are reading too many tombstones per 
request.

If you do have Tombstones without doing DELETE it because you probably TTL'ed 
the data when inserting (By mistake? Or did you set default_time_to_live in 
your table?). You can use nodetool cfstats to see how many tombstones per read 
slice you have. This is, probably, also the cause of your missing data. Data 
was tombstoned, so it is not available.



Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolohttp://linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.comhttp://www.pythian.com/

On Mon, Jun 15, 2015 at 10:54 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I have reloaded the data in my cluster of 3 nodes RF: 2.
I have loaded about 2 billion rows in one table.
I use LeveledCompactionStrategy on my table.
I use version 2.1.6.
I use the default cassandra.yaml, only the ip address for seeds and throughput 
has been change.

I loaded my data with simple insert statements. This took a bit more than one 
day to load the data… and one more day to compact the data on all nodes.
For me this is quite acceptable since I should not be doing this again.
I have done this with previous versions like 2.1.3 and others and I basically 
had absolutely no problems.

Now I read the log files on the client side, there I see no warning and no 
errors.
On the nodes side there I see many WARNING, all related with tombstones, but 
there are no ERRORS.

My problem is that I see some *many missing records* in the DB, and I have 
never observed this with previous versions.

1) Is this a know problem?
2) Do you have any idea how I could track down this problem?
3) What is the meaning of this WARNING (the only type of ERROR | WARN  I could 
find)?

WARN  [SharedPool-Worker-2] 2015-06-15 10:12:00,866 SliceQueryFilter.java:319 - 
Read 2990 live and 16016 tombstone cells in gttdata.alltrades_co_rep_pcode for 
key: D:07 (see tombstone_warn_threshold). 5000 columns were requested, 
slices=[388:201001-388:201412:!]


4) Is it possible to have Tombstone when we make no DELETE statements?

I’m lost…

Thanks for your help.



--

Re: Missing data

[Jean Tremblay]

Thanks Bryan.
I believe I have a different problem with the Datastax 2.1.6 driver.
My problem is not that I make huge selects.
My problem seems more to occur on some inserts. I inserts MANY rows and with 
the version 2.1.6 of the driver I seem to be loosing some records.

But thanks anyway I will remember your mail when I bump into the select problem.

Cheers

Jean


On 15 Jun 2015, at 19:13 , Bryan Holladay 
holla...@longsight.commailto:holla...@longsight.com wrote:

Theres your problem, you're using the DataStax java driver :) I just ran into 
this issue in the last week and it was incredibly frustrating. If you are doing 
a simple loop on a select *  query, then the DataStax java driver will only 
process 2^31 rows (e.g. the Java Integer Max (2,147,483,647)) before it stops 
w/o any error or output in the logs. The fact that you said you only had about 
2 billion rows but you are seeing missing data is a red flag.

I found the only way around this is to do your select * in chunks based on 
the token range (see this gist for an example: 
https://gist.github.com/baholladay/21eb4c61ea8905302195 )
Just loop for every 100million rows and make a new query select * from TABLE 
where token(key)  lastToken

Thanks,
Bryan




On Mon, Jun 15, 2015 at 12:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Dear all,

I identified a bit more closely the root cause of my missing data.

The problem is occurring when I use

dependency
groupIdcom.datastax.cassandra/groupId
artifactIdcassandra-driver-core/artifactId
version2.1.6/version
/dependency

on my client against Cassandra 2.1.6.

I did not have the problem when I was using the driver 2.1.4 with C* 2.1.4.
Interestingly enough I don’t have the problem with the driver 2.1.4 with C* 
2.1.6.  !!

So as far as I can locate the problem, I would say that the version 2.1.6 of 
the driver is not working properly and is loosing some of my records.!!!

——

As far as my tombstones are concerned I don’t understand their origin.
I removed all location in my code where I delete items, and I do not use TTL 
anywhere ( I don’t need this feature in my project).

And yet I have many tombstones building up.

Is there another origin for tombstone beside TTL, and deleting items? Could the 
compaction of LeveledCompactionStrategy be the origin of them?

@Carlos thanks for your guidance.

Kind regards

Jean



On 15 Jun 2015, at 11:17 , Carlos Rolo 
r...@pythian.commailto:r...@pythian.com wrote:

Hi Jean,

The problem of that Warning is that you are reading too many tombstones per 
request.

If you do have Tombstones without doing DELETE it because you probably TTL'ed 
the data when inserting (By mistake? Or did you set default_time_to_live in 
your table?). You can use nodetool cfstats to see how many tombstones per read 
slice you have. This is, probably, also the cause of your missing data. Data 
was tombstoned, so it is not available.



Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolohttp://linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 
x1649tel:%2B1%20613%20565%208696%20x1649
www.pythian.comhttp://www.pythian.com/

On Mon, Jun 15, 2015 at 10:54 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I have reloaded the data in my cluster of 3 nodes RF: 2.
I have loaded about 2 billion rows in one table.
I use LeveledCompactionStrategy on my table.
I use version 2.1.6.
I use the default cassandra.yaml, only the ip address for seeds and throughput 
has been change.

I loaded my data with simple insert statements. This took a bit more than one 
day to load the data… and one more day to compact the data on all nodes.
For me this is quite acceptable since I should not be doing this again.
I have done this with previous versions like 2.1.3 and others and I basically 
had absolutely no problems.

Now I read the log files on the client side, there I see no warning and no 
errors.
On the nodes side there I see many WARNING, all related with tombstones, but 
there are no ERRORS.

My problem is that I see some *many missing records* in the DB, and I have 
never observed this with previous versions.

1) Is this a know problem?
2) Do you have any idea how I could track down this problem?
3) What is the meaning of this WARNING (the only type of ERROR | WARN  I could 
find)?

WARN  [SharedPool-Worker-2] 2015-06-15 10:12:00,866 SliceQueryFilter.java:319 - 
Read 2990 live and 16016 tombstone cells in gttdata.alltrades_co_rep_pcode for 
key: D:07 (see tombstone_warn_threshold). 5000 columns were requested, 
slices=[388:201001-388:201412:!]


4) Is it possible to have Tombstone when we make no DELETE statements?

I’m lost…

Thanks for your help.



--

Re: cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for replica nodes' responses]

[Jean Tremblay]

I have experienced similar results: OperationTimedOut after inserting many 
millions of records on a 5 nodes cluster, using Cassandra 2.1.5.
I rolled back to 2.1.4 using identically the same configuration as with 2.1.5 
and these timeout went away… This is not the solution to your problem but just 
to say that for me 2.1.5 seems to saturate when bulk inserting many million 
records.


 On 28 May 2015, at 15:15 , Sachin PK sachinpray...@gmail.com wrote:
 
 Hi I'm running Cassandra 2.1.5 ,(single datacenter ,4 node,16GB vps each node 
 ),I have given my configuration below, I'm using python driver on my clients 
 ,when i tried to insert  1049067 items I got an error.
  
 cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for 
 replica nodes' responses] message=Operation timed out - received only 0 
 responses. info={'received_responses': 0, 'required_responses': 1, 
 'consistency': 'ONE'}
 
 also I'm getting an error when I check the count of CF from cqlsh
 
 OperationTimedOut: errors={}, last_host=127.0.0.1
 
 I've installed Datastax Ops center for monitoring  nodes , it shows about 
 1000/sec write request even the cluster is idle .
 
 Is there any problem with my cassandra configuration 
 
 
 cluster_name: 'Test Cluster'
 
 num_tokens: 256
 
 
 hinted_handoff_enabled: true
 .
 max_hint_window_in_ms: 1080 # 3 hours
 
 hinted_handoff_throttle_in_kb: 1024
 
 max_hints_delivery_threads: 2
 
 batchlog_replay_throttle_in_kb: 1024
 
 authenticator: AllowAllAuthenticator
 
 authorizer: AllowAllAuthorizer
 
 permissions_validity_in_ms: 2000
 
 partitioner: org.apache.cassandra.dht.Murmur3Partitioner
 
 data_file_directories:
 - /var/lib/cassandra/data
 
 commitlog_directory: /var/lib/cassandra/commitlog
 
 disk_failure_policy: stop
 
 commit_failure_policy: stop
 
 key_cache_size_in_mb:
 
 key_cache_save_period: 14400
 
 row_cache_size_in_mb: 0
 
 
 row_cache_save_period: 0
 
 counter_cache_size_in_mb:
 
 
 counter_cache_save_period: 7200
 
 
 saved_caches_directory: /var/lib/cassandra/saved_caches
 
 commitlog_sync: periodic
 commitlog_sync_period_in_ms: 1
 
 commitlog_segment_size_in_mb: 32
 
 seed_provider:
 
 - class_name: org.apache.cassandra.locator.SimpleSeedProvider
   parameters:
 
   - seeds: 10.xx.xx.xx,10.xx.xx.xxx
 
 concurrent_reads: 45
 concurrent_writes: 64
 concurrent_counter_writes: 32
 
 memtable_allocation_type: heap_buffers
 
 
 memtable_flush_writers: 6
 
 index_summary_capacity_in_mb:
 
 index_summary_resize_interval_in_minutes: 60
 
 trickle_fsync: false
 trickle_fsync_interval_in_kb: 10240
 
 storage_port: 7000
 
 ssl_storage_port: 7001
 
 listen_address: 10.xx.x.xxx
 
 start_native_transport: true
 
 native_transport_port: 9042
 
 rpc_address: 0.0.0.0
 
 rpc_port: 9160
 
 broadcast_rpc_address: 10.xx.x.xxx
 
 rpc_keepalive: true
 
 rpc_server_type: sync
 
 thrift_framed_transport_size_in_mb: 15
 
 incremental_backups: false
 
 snapshot_before_compaction: false
 
 auto_snapshot: true
 
 tombstone_warn_threshold: 1000
 tombstone_failure_threshold: 10
 
 column_index_size_in_kb: 64
 
 batch_size_warn_threshold_in_kb: 5
 
 compaction_throughput_mb_per_sec: 16
 
 sstable_preemptive_open_interval_in_mb: 50
 
 read_request_timeout_in_ms: 5000
 
 range_request_timeout_in_ms: 1
 
 write_request_timeout_in_ms: 2000
 
 counter_write_request_timeout_in_ms: 5000
 
 cas_contention_timeout_in_ms: 1000
 
 truncate_request_timeout_in_ms: 6
 
 request_timeout_in_ms: 1
 
 cross_node_timeout: false
 
 endpoint_snitch: SimpleSnitch
 
 
 dynamic_snitch_update_interval_in_ms: 100 
 
 dynamic_snitch_reset_interval_in_ms: 60
 
 dynamic_snitch_badness_threshold: 0.1
 
 request_scheduler: org.apache.cassandra.scheduler.NoScheduler
 
 
 server_encryption_options:
 internode_encryption: none
 keystore: conf/.keystore
 keystore_password: cassandra
 truststore: conf/.truststore
 truststore_password: cassandra
 
 
 # enable or disable client/server encryption.
 client_encryption_options:
 enabled: false
 keystore: conf/.keystore
 keystore_password: cassandra
 
 internode_compression: all
 
 inter_dc_tcp_nodelay: false
 
 
 
 

Re: LeveledCompactionStrategy

[Jean Tremblay]

I played around with these settings, namely the tombstone_threshold, and it 
**eventually** triggered a Tombstone Compaction.
Now I see that getting rid of these Tombstone is a process which takes some 
times.

I would like to be able to schedule a Tombstone Compaction.

Is there a way to trigger immediately a Tombstone Compaction on a table which 
is using LeveledCompactionStrategy?


Thanks a lot for your help

Jean

On 14 May 2015, at 22:45 , Nate McCall 
n...@thelastpickle.commailto:n...@thelastpickle.com wrote:

You can make LCS more aggressive with tombstone-only compactions via seting 
unchecked_tombstone_compaction=true and turn down tombstone_threshold to 0.05 
(maybe going up or down as needed). Details on both can be found here: 
http://docs.datastax.com/en/cql/3.1/cql/cql_reference/compactSubprop.html

As for monitoring tombstones, there is a tombstoneScannedHistogram on 
ColumnFamilyMetrics which measures how many tombstones were discarded during 
reads.

Also, you should take a couple of SSTables from production and use the 
sstablemetadata utility specifically looking at Estimated droppable 
tombstones and Estimated tombstone drop times output from such.

Spend some time experimenting with those settings incrementally. Finding the 
sweet spot is different for each workload will make a huge difference in 
overall performance.



On Thu, May 14, 2015 at 8:06 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:

 Hi,

 I’m using Cassandra 2.1.4 with a table using LeveledCompactionStrategy.
 Often I need to delete many rows and I want to make sure I don’t have too 
 many tombstones.

 How does one get rid of tombstones in a table using LCS?
 How can we monitor how many tombstones are around?

 Thanks for your help.

 Jean




--
-
Nate McCall
Austin, TX
@zznate

Co-Founder  Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.comhttp://www.thelastpickle.com/

Re: LeveledCompactionStrategy

[Jean Tremblay]

Thanks a lot.
On 14 May 2015, at 22:45 , Nate McCall 
n...@thelastpickle.commailto:n...@thelastpickle.com wrote:

You can make LCS more aggressive with tombstone-only compactions via seting 
unchecked_tombstone_compaction=true and turn down tombstone_threshold to 0.05 
(maybe going up or down as needed). Details on both can be found here: 
http://docs.datastax.com/en/cql/3.1/cql/cql_reference/compactSubprop.html

As for monitoring tombstones, there is a tombstoneScannedHistogram on 
ColumnFamilyMetrics which measures how many tombstones were discarded during 
reads.

Also, you should take a couple of SSTables from production and use the 
sstablemetadata utility specifically looking at Estimated droppable 
tombstones and Estimated tombstone drop times output from such.

Spend some time experimenting with those settings incrementally. Finding the 
sweet spot is different for each workload will make a huge difference in 
overall performance.



On Thu, May 14, 2015 at 8:06 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:

 Hi,

 I’m using Cassandra 2.1.4 with a table using LeveledCompactionStrategy.
 Often I need to delete many rows and I want to make sure I don’t have too 
 many tombstones.

 How does one get rid of tombstones in a table using LCS?
 How can we monitor how many tombstones are around?

 Thanks for your help.

 Jean




--
-
Nate McCall
Austin, TX
@zznate

Co-Founder  Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.comhttp://www.thelastpickle.com/

LeveledCompactionStrategy

[Jean Tremblay]

Hi,

I’m using Cassandra 2.1.4 with a table using LeveledCompactionStrategy.
Often I need to delete many rows and I want to make sure I don’t have too many 
tombstones.

How does one get rid of tombstones in a table using LCS?
How can we monitor how many tombstones are around?

Thanks for your help.

Jean

Re: How much disk is needed to compact Leveled compaction?

[Jean Tremblay]

I am only using LeveledCompactionStrategy, and as I describe in my original 
mail, I don’t understand why C* is complaining that it cannot compact when I 
have more than 40% free disk space.



On 07 Apr 2015, at 01:10 , Bryan Holladay 
holla...@longsight.commailto:holla...@longsight.com wrote:


What other storage impacting commands or nuances do you gave to consider when 
you switch to leveled compaction? For instance, nodetool cleanup says

Running the nodetool cleanup command causes a temporary increase in disk space 
usage proportional to the size of your largest SSTable.

Are sstables smaller with leveled compaction making this a non issue?

How can you determine what the new threshold for storage space is?

Thanks,
Bryan

On Apr 6, 2015 6:19 PM, DuyHai Doan 
doanduy...@gmail.commailto:doanduy...@gmail.com wrote:

If you have SSD, you may afford switching to leveled compaction strategy, which 
requires much less than 50% of the current dataset for free space

Le 5 avr. 2015 19:04, daemeon reiydelle 
daeme...@gmail.commailto:daeme...@gmail.com a écrit :

You appear to have multiple java binaries in your path. That needs to be 
resolved.

sent from my mobile
Daemeon C.M. Reiydelle
USA 415.501.0198tel:415.501.0198
London +44.0.20.8144.9872tel:%2B44.0.20.8144.9872

On Apr 5, 2015 1:40 AM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,
I have a cluster of 5 nodes. We use cassandra 2.1.3.

The 5 nodes use about 50-57% of the 1T SSD.
One node managed to compact all its data. During one compaction this node used 
almost 100% of the drive. The other nodes refuse to continue compaction 
claiming that there is not enough disk space.

From the documentation LeveledCompactionStrategy should be able to compact my 
data, well at least this is what I understand.

Size-tiered compaction requires at least as much free disk space for 
compaction as the size of the largest column family. Leveled compaction needs 
much less space for compaction, only 10 * sstable_size_in_mb. However, even if 
you’re using leveled compaction, you should leave much more free disk space 
available than this to accommodate streaming, repair, and snapshots, which can 
easily use 10GB or more of disk space. Furthermore, disk performance tends to 
decline after 80 to 90% of the disk space is used, so don’t push the 
boundaries.

This is the disk usage. Node 4 is the only one that could compact everything.
node0: /dev/disk1 931Gi 534Gi 396Gi 57% /
node1: /dev/disk1 931Gi 513Gi 417Gi 55% /
node2: /dev/disk1 931Gi 526Gi 404Gi 57% /
node3: /dev/disk1 931Gi 507Gi 424Gi 54% /
node4: /dev/disk1 931Gi 475Gi 456Gi 51% /

When I try to compact the other ones I get this:

objc[18698]: Class JavaLaunchHelper is implemented in both 
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java and 
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre/lib/libinstrument.dylib.
 One of the two will be used. Which one is undefined.
error: Not enough space for compaction, estimated sstables = 2894, expected 
write size = 485616651726
-- StackTrace --
java.lang.RuntimeException: Not enough space for compaction, estimated sstables 
= 2894, expected write size = 485616651726
at 
org.apache.cassandra.db.compaction.CompactionTask.checkAvailableDiskSpace(CompactionTask.java:293)
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:127)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:76)
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at 
org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:512)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

I did not set the sstable_size_in_mb I use the 160MB default.

Is it normal that during compaction it needs so much diskspace? What would be 
the best solution to overcome this problem?

Thanks for your help

Cassandra vs OS x

[Jean Tremblay]

Hi,

Why do everyone say that Cassandra should not be used in production on an Mac 
OS x?
Why would this not work?
Are there anyone out there using OS x in production? What is your experience 
with this?

Thanks

Jean

How much disk is needed to compact Leveled compaction?

[Jean Tremblay]

Hi,
I have a cluster of 5 nodes. We use cassandra 2.1.3.

The 5 nodes use about 50-57% of the 1T SSD.
One node managed to compact all its data. During one compaction this node used 
almost 100% of the drive. The other nodes refuse to continue compaction 
claiming that there is not enough disk space.

From the documentation LeveledCompactionStrategy should be able to compact my 
data, well at least this is what I understand.

Size-tiered compaction requires at least as much free disk space for 
compaction as the size of the largest column family. Leveled compaction needs 
much less space for compaction, only 10 * sstable_size_in_mb. However, even if 
you’re using leveled compaction, you should leave much more free disk space 
available than this to accommodate streaming, repair, and snapshots, which can 
easily use 10GB or more of disk space. Furthermore, disk performance tends to 
decline after 80 to 90% of the disk space is used, so don’t push the 
boundaries.

This is the disk usage. Node 4 is the only one that could compact everything.
node0: /dev/disk1 931Gi 534Gi 396Gi 57% /
node1: /dev/disk1 931Gi 513Gi 417Gi 55% /
node2: /dev/disk1 931Gi 526Gi 404Gi 57% /
node3: /dev/disk1 931Gi 507Gi 424Gi 54% /
node4: /dev/disk1 931Gi 475Gi 456Gi 51% /

When I try to compact the other ones I get this:

objc[18698]: Class JavaLaunchHelper is implemented in both 
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java and 
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre/lib/libinstrument.dylib.
 One of the two will be used. Which one is undefined.
error: Not enough space for compaction, estimated sstables = 2894, expected 
write size = 485616651726
-- StackTrace --
java.lang.RuntimeException: Not enough space for compaction, estimated sstables 
= 2894, expected write size = 485616651726
at 
org.apache.cassandra.db.compaction.CompactionTask.checkAvailableDiskSpace(CompactionTask.java:293)
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:127)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:76)
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at 
org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:512)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

I did not set the sstable_size_in_mb I use the 160MB default.

Is it normal that during compaction it needs so much diskspace? What would be 
the best solution to overcome this problem?

Thanks for your help