Re: full gc too often

[Jonathan Haddad]

I recommend reading through
https://issues.apache.org/jira/browse/CASSANDRA-8150 to get an idea of how
the JVM GC works and what you can do to tune it.  Also good is Blake
Eggleston's writeup which can be found here:
http://blakeeggleston.com/cassandra-tuning-the-jvm-for-read-heavy-workloads.html

I'd like to note that allocating 4GB heap to Cassandra under any serious
workload is unlikely to be sufficient.


On Thu Dec 04 2014 at 8:43:38 PM Philo Yang  wrote:

> I have two kinds of machine:
> 16G RAM, with default heap size setting, about 4G.
> 64G RAM, with default heap size setting, about 8G.
>
> These two kinds of nodes have same number of vnodes, and both of them have
> gc issue, although the node of 16G have a higher probability  of gc issue.
>
> Thanks,
> Philo Yang
>
>
> 2014-12-05 12:34 GMT+08:00 Tim Heckman :
>
>> On Dec 4, 2014 8:14 PM, "Philo Yang"  wrote:
>> >
>> > Hi,all
>> >
>> > I have a cluster on C* 2.1.1 and jdk 1.7_u51. I have a trouble with
>> full gc that sometime there may be one or two nodes full gc more than one
>> time per minute and over 10 seconds each time, then the node will be
>> unreachable and the latency of cluster will be increased.
>> >
>> > I grep the GCInspector's log, I found when the node is running fine
>> without gc trouble there are two kinds of gc:
>> > ParNew GC in less than 300ms which clear the Par Eden Space and
>> enlarge CMS Old Gen/ Par Survivor Space little (because it only show gc in
>> more than 200ms, there is only a small number of ParNew GC in log)
>> > ConcurrentMarkSweep in 4000~8000ms which reduce CMS Old Gen much and
>> enlarge Par Eden Space little, each 1-2 hours it will be executed once.
>> >
>> > However, sometimes ConcurrentMarkSweep will be strange like it shows:
>> >
>> > INFO  [Service Thread] 2014-12-05 11:28:44,629 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 12648ms.  CMS Old Gen: 3579838424 -> 3579838464;
>> Par Eden Space: 503316480 -> 294794576; Par Survivor Space: 62914528 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:28:59,581 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 12227ms.  CMS Old Gen: 3579838464 -> 3579836512;
>> Par Eden Space: 503316480 -> 310562032; Par Survivor Space: 62872496 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:29:14,686 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 11538ms.  CMS Old Gen: 3579836688 -> 3579805792;
>> Par Eden Space: 503316480 -> 332391096; Par Survivor Space: 62914544 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:29:29,371 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 12180ms.  CMS Old Gen: 3579835784 -> 3579829760;
>> Par Eden Space: 503316480 -> 351991456; Par Survivor Space: 62914552 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:29:45,028 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 10574ms.  CMS Old Gen: 3579838112 -> 3579799752;
>> Par Eden Space: 503316480 -> 366222584; Par Survivor Space: 62914560 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:29:59,546 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 11594ms.  CMS Old Gen: 3579831424 -> 3579817392;
>> Par Eden Space: 503316480 -> 388702928; Par Survivor Space: 62914552 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:30:14,153 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 11463ms.  CMS Old Gen: 3579817392 -> 3579838424;
>> Par Eden Space: 503316480 -> 408992784; Par Survivor Space: 62896720 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:30:25,009 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 9576ms.  CMS Old Gen: 3579838424 -> 3579816424;
>> Par Eden Space: 503316480 -> 438633608; Par Survivor Space: 62914544 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:30:39,929 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 11556ms.  CMS Old Gen: 3579816424 -> 3579785496;
>> Par Eden Space: 503316480 -> 441354856; Par Survivor Space: 62889528 -> 0
>> > INFO  [Service Thread] 2014-12-05 11:30:54,085 GCInspector.java:142 -
>> ConcurrentMarkSweep GC in 12082ms.  CMS Old Gen: 3579786592 -> 3579814464;
>> Par Eden Space: 503316480 -> 448782440; Par Survivor Space: 62914560 -> 0
>> >
>> > In each time Old Gen reduce only a little, Survivor Space will be clear
>> but the heap is still full so there will be another full gc very soon then
>> the node will down. If I restart the node, it will be fine without gc
>> trouble.
>> >
>> > Can anyone help me to find out where is the problem that full gc can't
>> reduce CMS Old Gen? Is it because there are too many objects in heap can't
>> be recycled? I think review the table scheme designing and add new nodes
>> into cluster is a good idea, but I still want to know if there is any other
>> reason causing this trouble.
>>
>> How much total system memory do you have? How much is allocated for heap
>> usage? How big is your working data set?
>>
>> The reason I ask is that I've seen problems with lots of GC with no room
>> gained, and it was memory pressure. Not enough for the heap. We decided
>> that just increasing the heap size was a bad idea, as we did rely

Re: Performance Difference between Batch Insert and Bulk Load

[Dong Dai]

> On Dec 4, 2014, at 1:46 PM, Tyler Hobbs  wrote:
> 
> 
> On Thu, Dec 4, 2014 at 11:50 AM, Dong Dai  > wrote:
> As we already did what coordinators do in client side, why don’t we do one 
> step more:
> break the UNLOGGED batch statements into several small batch statements, each 
> of which contains
> the statements with the same partition key. And send them to different 
> coordinators based
> on TokenAwarePolicy? This will save lots of RPC times, right?
> 
> The reason I asked is I have a use case where importing huge data into 
> Cassandra is a very common case, and all these importing do not need to be 
> atomic.
> 
> Yes, what you suggest is basically ideal.  I would do exactly that.
> 

Sounds great! By the way, will you create a ticket for this, so we can follow 
the updates?

thanks,
- Dong

> 
> -- 
> Tyler Hobbs
> DataStax 

Re: Replacing a dead node by deleting it and auto_bootstrap'ing a new node (Cassandra 2.0)

[Jaydeep Chovatia]

as per my knowledge if you have externally NOT specified
"-Dcassandra.replace_address=old_node_ipaddress" then new tokens (randomly)
would get assigned to bootstrapping node instead of tokens of dead node.

-jaydeep

On Thu, Dec 4, 2014 at 6:50 AM, Omri Bahumi  wrote:

> Hi,
>
> I was wondering, how would auto_bootstrap behave in this scenario:
>
> 1. I had a cluster with 3 nodes (RF=2)
> 2. One node died, I deleted it with "nodetool removenode" (+ force)
> 3. A new node launched with "auto_bootstrap: true"
>
> The question is: will the "right" vnodes go to the new node as if it
> was bootstrapped with "-Dcassandra.replace_address=old_node_ipaddress"
> ?
>
> Thanks,
> Omri.
>

Re: full gc too often

[Philo Yang]

I have two kinds of machine:
16G RAM, with default heap size setting, about 4G.
64G RAM, with default heap size setting, about 8G.

These two kinds of nodes have same number of vnodes, and both of them have
gc issue, although the node of 16G have a higher probability  of gc issue.

Thanks,
Philo Yang


2014-12-05 12:34 GMT+08:00 Tim Heckman :

> On Dec 4, 2014 8:14 PM, "Philo Yang"  wrote:
> >
> > Hi,all
> >
> > I have a cluster on C* 2.1.1 and jdk 1.7_u51. I have a trouble with full
> gc that sometime there may be one or two nodes full gc more than one time
> per minute and over 10 seconds each time, then the node will be unreachable
> and the latency of cluster will be increased.
> >
> > I grep the GCInspector's log, I found when the node is running fine
> without gc trouble there are two kinds of gc:
> > ParNew GC in less than 300ms which clear the Par Eden Space and
> enlarge CMS Old Gen/ Par Survivor Space little (because it only show gc in
> more than 200ms, there is only a small number of ParNew GC in log)
> > ConcurrentMarkSweep in 4000~8000ms which reduce CMS Old Gen much and
> enlarge Par Eden Space little, each 1-2 hours it will be executed once.
> >
> > However, sometimes ConcurrentMarkSweep will be strange like it shows:
> >
> > INFO  [Service Thread] 2014-12-05 11:28:44,629 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 12648ms.  CMS Old Gen: 3579838424 -> 3579838464;
> Par Eden Space: 503316480 -> 294794576; Par Survivor Space: 62914528 -> 0
> > INFO  [Service Thread] 2014-12-05 11:28:59,581 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 12227ms.  CMS Old Gen: 3579838464 -> 3579836512;
> Par Eden Space: 503316480 -> 310562032; Par Survivor Space: 62872496 -> 0
> > INFO  [Service Thread] 2014-12-05 11:29:14,686 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 11538ms.  CMS Old Gen: 3579836688 -> 3579805792;
> Par Eden Space: 503316480 -> 332391096; Par Survivor Space: 62914544 -> 0
> > INFO  [Service Thread] 2014-12-05 11:29:29,371 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 12180ms.  CMS Old Gen: 3579835784 -> 3579829760;
> Par Eden Space: 503316480 -> 351991456; Par Survivor Space: 62914552 -> 0
> > INFO  [Service Thread] 2014-12-05 11:29:45,028 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 10574ms.  CMS Old Gen: 3579838112 -> 3579799752;
> Par Eden Space: 503316480 -> 366222584; Par Survivor Space: 62914560 -> 0
> > INFO  [Service Thread] 2014-12-05 11:29:59,546 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 11594ms.  CMS Old Gen: 3579831424 -> 3579817392;
> Par Eden Space: 503316480 -> 388702928; Par Survivor Space: 62914552 -> 0
> > INFO  [Service Thread] 2014-12-05 11:30:14,153 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 11463ms.  CMS Old Gen: 3579817392 -> 3579838424;
> Par Eden Space: 503316480 -> 408992784; Par Survivor Space: 62896720 -> 0
> > INFO  [Service Thread] 2014-12-05 11:30:25,009 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 9576ms.  CMS Old Gen: 3579838424 -> 3579816424;
> Par Eden Space: 503316480 -> 438633608; Par Survivor Space: 62914544 -> 0
> > INFO  [Service Thread] 2014-12-05 11:30:39,929 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 11556ms.  CMS Old Gen: 3579816424 -> 3579785496;
> Par Eden Space: 503316480 -> 441354856; Par Survivor Space: 62889528 -> 0
> > INFO  [Service Thread] 2014-12-05 11:30:54,085 GCInspector.java:142 -
> ConcurrentMarkSweep GC in 12082ms.  CMS Old Gen: 3579786592 -> 3579814464;
> Par Eden Space: 503316480 -> 448782440; Par Survivor Space: 62914560 -> 0
> >
> > In each time Old Gen reduce only a little, Survivor Space will be clear
> but the heap is still full so there will be another full gc very soon then
> the node will down. If I restart the node, it will be fine without gc
> trouble.
> >
> > Can anyone help me to find out where is the problem that full gc can't
> reduce CMS Old Gen? Is it because there are too many objects in heap can't
> be recycled? I think review the table scheme designing and add new nodes
> into cluster is a good idea, but I still want to know if there is any other
> reason causing this trouble.
>
> How much total system memory do you have? How much is allocated for heap
> usage? How big is your working data set?
>
> The reason I ask is that I've seen problems with lots of GC with no room
> gained, and it was memory pressure. Not enough for the heap. We decided
> that just increasing the heap size was a bad idea, as we did rely on free
> RAM being used for filesystem caching. So some vertical and horizontal
> scaling allowed us to give Cass more heap space, as well as distribute the
> workload to try and avoid further problems.
>
> > Thanks,
> > Philo Yang
>
> Cheers!
> -Tim
>

Re: full gc too often

[Tim Heckman]

On Dec 4, 2014 8:14 PM, "Philo Yang"  wrote:
>
> Hi,all
>
> I have a cluster on C* 2.1.1 and jdk 1.7_u51. I have a trouble with full
gc that sometime there may be one or two nodes full gc more than one time
per minute and over 10 seconds each time, then the node will be unreachable
and the latency of cluster will be increased.
>
> I grep the GCInspector's log, I found when the node is running fine
without gc trouble there are two kinds of gc:
> ParNew GC in less than 300ms which clear the Par Eden Space and
enlarge CMS Old Gen/ Par Survivor Space little (because it only show gc in
more than 200ms, there is only a small number of ParNew GC in log)
> ConcurrentMarkSweep in 4000~8000ms which reduce CMS Old Gen much and
enlarge Par Eden Space little, each 1-2 hours it will be executed once.
>
> However, sometimes ConcurrentMarkSweep will be strange like it shows:
>
> INFO  [Service Thread] 2014-12-05 11:28:44,629 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12648ms.  CMS Old Gen: 3579838424 -> 3579838464;
Par Eden Space: 503316480 -> 294794576; Par Survivor Space: 62914528 -> 0
> INFO  [Service Thread] 2014-12-05 11:28:59,581 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12227ms.  CMS Old Gen: 3579838464 -> 3579836512;
Par Eden Space: 503316480 -> 310562032; Par Survivor Space: 62872496 -> 0
> INFO  [Service Thread] 2014-12-05 11:29:14,686 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11538ms.  CMS Old Gen: 3579836688 -> 3579805792;
Par Eden Space: 503316480 -> 332391096; Par Survivor Space: 62914544 -> 0
> INFO  [Service Thread] 2014-12-05 11:29:29,371 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12180ms.  CMS Old Gen: 3579835784 -> 3579829760;
Par Eden Space: 503316480 -> 351991456; Par Survivor Space: 62914552 -> 0
> INFO  [Service Thread] 2014-12-05 11:29:45,028 GCInspector.java:142 -
ConcurrentMarkSweep GC in 10574ms.  CMS Old Gen: 3579838112 -> 3579799752;
Par Eden Space: 503316480 -> 366222584; Par Survivor Space: 62914560 -> 0
> INFO  [Service Thread] 2014-12-05 11:29:59,546 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11594ms.  CMS Old Gen: 3579831424 -> 3579817392;
Par Eden Space: 503316480 -> 388702928; Par Survivor Space: 62914552 -> 0
> INFO  [Service Thread] 2014-12-05 11:30:14,153 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11463ms.  CMS Old Gen: 3579817392 -> 3579838424;
Par Eden Space: 503316480 -> 408992784; Par Survivor Space: 62896720 -> 0
> INFO  [Service Thread] 2014-12-05 11:30:25,009 GCInspector.java:142 -
ConcurrentMarkSweep GC in 9576ms.  CMS Old Gen: 3579838424 -> 3579816424;
Par Eden Space: 503316480 -> 438633608; Par Survivor Space: 62914544 -> 0
> INFO  [Service Thread] 2014-12-05 11:30:39,929 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11556ms.  CMS Old Gen: 3579816424 -> 3579785496;
Par Eden Space: 503316480 -> 441354856; Par Survivor Space: 62889528 -> 0
> INFO  [Service Thread] 2014-12-05 11:30:54,085 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12082ms.  CMS Old Gen: 3579786592 -> 3579814464;
Par Eden Space: 503316480 -> 448782440; Par Survivor Space: 62914560 -> 0
>
> In each time Old Gen reduce only a little, Survivor Space will be clear
but the heap is still full so there will be another full gc very soon then
the node will down. If I restart the node, it will be fine without gc
trouble.
>
> Can anyone help me to find out where is the problem that full gc can't
reduce CMS Old Gen? Is it because there are too many objects in heap can't
be recycled? I think review the table scheme designing and add new nodes
into cluster is a good idea, but I still want to know if there is any other
reason causing this trouble.

How much total system memory do you have? How much is allocated for heap
usage? How big is your working data set?

The reason I ask is that I've seen problems with lots of GC with no room
gained, and it was memory pressure. Not enough for the heap. We decided
that just increasing the heap size was a bad idea, as we did rely on free
RAM being used for filesystem caching. So some vertical and horizontal
scaling allowed us to give Cass more heap space, as well as distribute the
workload to try and avoid further problems.

> Thanks,
> Philo Yang

Cheers!
-Tim

full gc too often

[Philo Yang]

Hi,all

I have a cluster on C* 2.1.1 and jdk 1.7_u51. I have a trouble with full gc
that sometime there may be one or two nodes full gc more than one time per
minute and over 10 seconds each time, then the node will be unreachable and
the latency of cluster will be increased.

I grep the GCInspector's log, I found when the node is running fine without
gc trouble there are two kinds of gc:
ParNew GC in less than 300ms which clear the Par Eden Space and enlarge CMS
Old Gen/ Par Survivor Space little (because it only show gc in more than
200ms, there is only a small number of ParNew GC in log)
ConcurrentMarkSweep in 4000~8000ms which reduce CMS Old Gen much and
enlarge Par Eden Space little, each 1-2 hours it will be executed once.

However, sometimes ConcurrentMarkSweep will be strange like it shows:

INFO  [Service Thread] 2014-12-05 11:28:44,629 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12648ms.  CMS Old Gen: 3579838424 -> 3579838464;
Par Eden Space: 503316480 -> 294794576; Par Survivor Space: 62914528 -> 0
INFO  [Service Thread] 2014-12-05 11:28:59,581 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12227ms.  CMS Old Gen: 3579838464 -> 3579836512;
Par Eden Space: 503316480 -> 310562032; Par Survivor Space: 62872496 -> 0
INFO  [Service Thread] 2014-12-05 11:29:14,686 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11538ms.  CMS Old Gen: 3579836688 -> 3579805792;
Par Eden Space: 503316480 -> 332391096; Par Survivor Space: 62914544 -> 0
INFO  [Service Thread] 2014-12-05 11:29:29,371 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12180ms.  CMS Old Gen: 3579835784 -> 3579829760;
Par Eden Space: 503316480 -> 351991456; Par Survivor Space: 62914552 -> 0
INFO  [Service Thread] 2014-12-05 11:29:45,028 GCInspector.java:142 -
ConcurrentMarkSweep GC in 10574ms.  CMS Old Gen: 3579838112 -> 3579799752;
Par Eden Space: 503316480 -> 366222584; Par Survivor Space: 62914560 -> 0
INFO  [Service Thread] 2014-12-05 11:29:59,546 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11594ms.  CMS Old Gen: 3579831424 -> 3579817392;
Par Eden Space: 503316480 -> 388702928; Par Survivor Space: 62914552 -> 0
INFO  [Service Thread] 2014-12-05 11:30:14,153 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11463ms.  CMS Old Gen: 3579817392 -> 3579838424;
Par Eden Space: 503316480 -> 408992784; Par Survivor Space: 62896720 -> 0
INFO  [Service Thread] 2014-12-05 11:30:25,009 GCInspector.java:142 -
ConcurrentMarkSweep GC in 9576ms.  CMS Old Gen: 3579838424 -> 3579816424;
Par Eden Space: 503316480 -> 438633608; Par Survivor Space: 62914544 -> 0
INFO  [Service Thread] 2014-12-05 11:30:39,929 GCInspector.java:142 -
ConcurrentMarkSweep GC in 11556ms.  CMS Old Gen: 3579816424 -> 3579785496;
Par Eden Space: 503316480 -> 441354856; Par Survivor Space: 62889528 -> 0
INFO  [Service Thread] 2014-12-05 11:30:54,085 GCInspector.java:142 -
ConcurrentMarkSweep GC in 12082ms.  CMS Old Gen: 3579786592 -> 3579814464;
Par Eden Space: 503316480 -> 448782440; Par Survivor Space: 62914560 -> 0

In each time Old Gen reduce only a little, Survivor Space will be clear but
the heap is still full so there will be another full gc very soon then the
node will down. If I restart the node, it will be fine without gc trouble.

Can anyone help me to find out where is the problem that full gc can't
reduce CMS Old Gen? Is it because there are too many objects in heap can't
be recycled? I think review the table scheme designing and add new nodes
into cluster is a good idea, but I still want to know if there is any other
reason causing this trouble.

Thanks,
Philo Yang

Re: Performance Difference between Batch Insert and Bulk Load

[Shane Hansen]

I'd be really interested to know what sort of performance or load
improvements you see by
doing client side partitioning. Please post back some results if you've
tried that strategy.

On Thu, Dec 4, 2014 at 11:46 AM, Tyler Hobbs  wrote:

>
> On Thu, Dec 4, 2014 at 11:50 AM, Dong Dai  wrote:
>
>> As we already did what coordinators do in client side, why don’t we do
>> one step more:
>> break the UNLOGGED batch statements into several small batch statements,
>> each of which contains
>> the statements with the same partition key. And send them to different
>> coordinators based
>> on TokenAwarePolicy? This will save lots of RPC times, right?
>>
>> The reason I asked is I have a use case where importing huge data into
>> Cassandra is a very common case, and all these importing do not need to
>> be atomic.
>>
>
> Yes, what you suggest is basically ideal.  I would do exactly that.
>
>
> --
> Tyler Hobbs
> DataStax 
>

Re: Performance Difference between Batch Insert and Bulk Load

[Tyler Hobbs]

On Thu, Dec 4, 2014 at 11:50 AM, Dong Dai  wrote:

> As we already did what coordinators do in client side, why don’t we do one
> step more:
> break the UNLOGGED batch statements into several small batch statements,
> each of which contains
> the statements with the same partition key. And send them to different
> coordinators based
> on TokenAwarePolicy? This will save lots of RPC times, right?
>
> The reason I asked is I have a use case where importing huge data into
> Cassandra is a very common case, and all these importing do not need to be
> atomic.
>

Yes, what you suggest is basically ideal.  I would do exactly that.


-- 
Tyler Hobbs
DataStax 

Re: Performance Difference between Batch Insert and Bulk Load

[Dong Dai]

> On Dec 4, 2014, at 11:37 AM, Tyler Hobbs  wrote:
> 
> 
> On Wed, Dec 3, 2014 at 11:02 PM, Dong Dai  > wrote:
> 
> 1) except I am using TokenAwarePolicy, the async insert also can not be sent 
> to 
> the right coordinator. 
> 
> Yes.  Of course, TokenAwarePolicy can wrap any other policy.
>  
> 
> 2) the TokenAwarePolicy actually is doing the job that coordinators
> do: calculate the data placement by the keyspace and partition key. 
> 
> That's correct, it does the same calculation that the coordinator does.
> 

Thanks for the clarification. This leads to my previous discussion with Ryan. 
As we already did what coordinators do in client side, why don’t we do one step 
more:
break the UNLOGGED batch statements into several small batch statements, each 
of which contains
the statements with the same partition key. And send them to different 
coordinators based
on TokenAwarePolicy? This will save lots of RPC times, right?

The reason I asked is I have a use case where importing huge data into 
Cassandra is a very common case, and all these importing do not need to be 
atomic.

thanks,
- Dong

> 
> -- 
> Tyler Hobbs
> DataStax 

Re: mirror between github and apache

[Tyler Hobbs]

The Apache git repo is the main repo.  The github repo is periodically
synched (I believe every few hours).

On Thu, Dec 4, 2014 at 2:39 AM, Christian Andersson 
wrote:

> Hi,
>
> Have a question regarding the mirror of git://git.apache.org/cassandra.git
> and github.
>
> How are the repositories synchronized? Manually, some kind of automatic
> job doing this..
>
> //chibbe
>



-- 
Tyler Hobbs
DataStax 

Re: Performance Difference between Batch Insert and Bulk Load

[Tyler Hobbs]

On Wed, Dec 3, 2014 at 11:02 PM, Dong Dai  wrote:

>
> 1) except I am using TokenAwarePolicy, the async insert also can not be
> sent to
> the right coordinator.
>

Yes.  Of course, TokenAwarePolicy can wrap any other policy.


>
> 2) the TokenAwarePolicy actually is doing the job that coordinators
> do: calculate the data placement by the keyspace and partition key.
>

That's correct, it does the same calculation that the coordinator does.


-- 
Tyler Hobbs
DataStax 

unsubscribe

[Sheausong Yang]

>

Repair taking many snapshots per minute

[Robert Wille]

This is a follow-up to my previous post “Cassandra taking snapshots 
automatically?”. I’ve renamed the thread to better describe the new information 
I’ve discovered.

I have a four node, RF=3, 2.0.11 cluster that was producing snapshots at a 
prodigious rate. I let the cluster sit idle overnight to settle down, and 
deleted all the snapshots. I waited for a while to make sure it really was done 
creating snapshots. I then ran "nodetool repair test2_browse” on one node and 
immediately got snapshots on three of my four nodes. Here’s what my 
/var/lib/cassandra/data/test2_browse/path_by_parent/snapshots directory looks 
like after a few minutes:

drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
33adb6b0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
33aea110-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
33af6460-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
33b027b0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
33b0c3f0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
33b1ae50-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
33b24a90-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
3a2d1300-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
3a2daf40-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
3a2e4b80-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
3a2ee7c0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
3a2f5cf0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:09 
3a2ff930-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
40bbb190-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
40bc74e0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
40bd1120-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
40bdd470-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
40be70b0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
474b3a80-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
474c24e0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
474d3650-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
4dd9d910-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
4ddac370-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
546877a0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
54696200-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
546a7370-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
546b36c0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:10 
5af73d40-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
60ee7dd0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
60ef4120-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
60efdd60-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
60f0a0b0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
60f16400-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
677d1c60-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
677e06c0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
6e0bbaf0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
749a5980-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
7b28f810-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:11 
81b796a0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:12 
87ae3af0-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:12 
87aed730-7bbf-11e4-893d-d96c3e745723
drwxr-xr-x 2 cassandra cassandra 40960 Dec  4 07:12 
87af7370-7bbf-11e4-893d-d96c3e745723

I also get lots of events like these in system.log:

ERROR [AntiEntropySessions:1] 2014-12-03 13:35:40,541 CassandraDaemon.java 
(line 199) Exception in thread Thread[AntiEntropySessions:1,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot 
creation.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.F

Replacing a dead node by deleting it and auto_bootstrap'ing a new node (Cassandra 2.0)

[Omri Bahumi]

Hi,

I was wondering, how would auto_bootstrap behave in this scenario:

1. I had a cluster with 3 nodes (RF=2)
2. One node died, I deleted it with "nodetool removenode" (+ force)
3. A new node launched with "auto_bootstrap: true"

The question is: will the "right" vnodes go to the new node as if it
was bootstrapped with "-Dcassandra.replace_address=old_node_ipaddress"
?

Thanks,
Omri.

Re: [Import csv to Cassandra] Taking too much time

[Yuki Morishita]

Here's blog post about writing SSTables from CSV and using
SSTableLoader to load them.

http://www.datastax.com/dev/blog/using-the-cassandra-bulk-loader-updated

On Thu, Dec 4, 2014 at 5:57 AM, 严超  wrote:
> Thank you very much for your advice.
> Can you give me more advice for using SSTableLoader to import csv ?
> What is the best practice to use SStableLoader  importing csv in Cassandra ?
>
> Best Regards！
> Chao Yan
> --
> My twitter：Andy Yan @yanchao727
> My Weibo：http://weibo.com/herewearenow
> --
>
> 2014-12-04 18:58 GMT+08:00 Akshay Ballarpure :
>>
>> Hello Chao Yan,
>> CSV data import using Copy command in cassandra is always painful for
>> large size file (say > 1Gig).
>> CQL tool is not developed for performing such heavy operations instead try
>> using SSTableLoader to import.
>>
>>
>> Best Regards
>> Akshay
>>
>>
>>
>> From:严超 
>> To:user@cassandra.apache.org
>> Date:12/04/2014 02:11 PM
>> Subject:[Import csv to Cassandra] Taking too much time
>> 
>>
>>
>>
>> Hi, Everyone:
>> I'm importing a CSV file into Cassandra, and it always get
>> error:"Request did not complete within rpc_timeout ", then I have to
>> continue my COPY command of cql again. And the CSV file is 2.2 G . It is
>> taking a long time.
>> How can I speed up csv file importing ? Is there another way
>> except COPY command of cql ?
>> Thank you for any help .
>>
>> Best Regards！
>> Chao Yan
>> --
>> My twitter：Andy Yan @yanchao727
>> My Weibo：http://weibo.com/herewearenow
>> --
>>
>> =-=-=
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>
>



-- 
Yuki Morishita
 t:yukim (http://twitter.com/yukim)

Re: [Import csv to Cassandra] Taking too much time

[严超]

Thank you very much for your advice.
Can you give me more advice for using SSTableLoader to import csv ?
What is the best practice to use SStableLoader  importing csv in Cassandra ?

*Best Regards！*


*Chao Yan--**My twitter：Andy Yan @yanchao727
*


*My Weibo：http://weibo.com/herewearenow
--*

2014-12-04 18:58 GMT+08:00 Akshay Ballarpure :

> Hello Chao Yan,
> CSV data import using Copy command in cassandra is always painful for
> large size file (say > 1Gig).
> CQL tool is not developed for performing such heavy operations instead try
> using SSTableLoader to import.
>
>
> Best Regards
> Akshay
>
>
>
> From:严超 
> To:user@cassandra.apache.org
> Date:12/04/2014 02:11 PM
> Subject:[Import csv to Cassandra] Taking too much time
> --
>
>
>
> Hi, Everyone:
> I'm importing a CSV file into Cassandra, and it always get
> error:"Request did not complete within rpc_timeout ", then I have to
> continue my COPY command of cql again. And the CSV file is 2.2 G . It is
> taking a long time.
> How can I speed up csv file importing ? Is there another way
> except COPY command of cql ?
> Thank you for any help .
>
> *Best Regards！*
>
>
> *Chao Yan -- My twitter：Andy Yan **@yanchao727*
> 
> * My Weibo：**http://weibo.com/herewearenow*
> 
> * --*
>
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>

Re: [Import csv to Cassandra] Taking too much time

[Akshay Ballarpure]

Hello Chao Yan,
CSV data import using Copy command in cassandra is always painful for 
large size file (say > 1Gig).
CQL tool is not developed for performing such heavy operations instead try 
using SSTableLoader to import.


Best Regards
Akshay 



From:   严超 
To: user@cassandra.apache.org
Date:   12/04/2014 02:11 PM
Subject:[Import csv to Cassandra] Taking too much time



Hi, Everyone:
I'm importing a CSV file into Cassandra, and it always get 
error:"Request did not complete within rpc_timeout ", then I have to 
continue my COPY command of cql again. And the CSV file is 2.2 G . It is 
taking a long time.
How can I speed up csv file importing ? Is there another way 
except COPY command of cql ?
Thank you for any help .

Best Regards！
Chao Yan
--
My twitter：Andy Yan @yanchao727
My Weibo：http://weibo.com/herewearenow
--

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: 2.0.10 upgrade to 2.1.2 gives "Unable to gossip with any seeds"

[Neha]

Check if u have rpc_server = hsha .. Change it to sync and try ..

Sent from my iPhone 

On Dec 4, 2014, at 3:55 PM, sinonim  wrote:

Hi all,

We have the case of a cassandra cluster with nodes version 2.0.10, all in a 
single EC2 region. We want to perform a rolling upgrade to version 2.1.2 but 
the new node has the following exceptions:

java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1222) 
~[apache-cassandra-2.1.2.jar:2.1.2]

Any ideas? The listen, rpc and seed addresses are the right ones.

Thank you.

2.0.10 upgrade to 2.1.2 gives "Unable to gossip with any seeds"

[sinonim]

Hi all,

We have the case of a cassandra cluster with nodes version 2.0.10, all in a
single EC2 region. We want to perform a rolling upgrade to version 2.1.2
but the new node has the following exceptions:

java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1222)
~[apache-cassandra-2.1.2.jar:2.1.2]

Any ideas? The listen, rpc and seed addresses are the right ones.

Thank you.

Re: Wide rows best practices and GC impact

[Jabbar Azam]

Hello,

I saw this earlier yesterday but didn't want to reply because I didn't know
what the cause was.

Basically I using wide rows with cassandra 1.x and was inserting data
constantly. After about 18 hours the JVM would crash with a dump file. For
some reason I removed the compaction throttling and the problem
disappeared. I've never really found out what the root cause was.


On Thu Dec 04 2014 at 2:49:57 AM Gianluca Borello 
wrote:

> Thanks Robert, I really appreciate your help!
>
> I'm still unsure why Cassandra 2.1 seem to perform much better in that
> same scenario (even setting the same values of compaction threshold and
> number of compactors), but I guess we'll revise when we'll decide to
> upgrade 2.1 in production.
>
> On Dec 3, 2014 6:33 PM, "Robert Coli"  wrote:
> >
> > On Tue, Dec 2, 2014 at 5:01 PM, Gianluca Borello 
> wrote:
> >>
> >> We mainly store time series-like data, where each data point is a
> binary blob of 5-20KB. We use wide rows, and try to put in the same row all
> the data that we usually need in a single query (but not more than that).
> As a result, our application logic is very simple (since we have to do just
> one query to read the data on average) and read/write response times are
> very satisfactory. This is a cfhistograms and a cfstats of our heaviest CF:
> >
> >
> > 100mb is not HYOOOGE but is around the size where large rows can cause
> heap pressure.
> >
> > You seem to be unclear on the implications of pending compactions,
> however.
> >
> > Briefly, pending compactions indicate that you have more SSTables than
> you "should". As compaction both merges row versions and reduces the number
> of SSTables, a high number of pending compactions causes problems
> associated with both having too many row versions ("fragmentation") and a
> large number of SSTables (per-SSTable heap/memory (depending on version)
> overhead like bloom filters and index samples). In your case, it seems the
> problem is probably just the compaction throttle being too low.
> >
> > My conjecture is that, given your normal data size and read/write
> workload, you are relatively close to "GC pre-fail" when compaction is
> working. When it stops working, you relatively quickly get into a state
> where you exhaust heap because you have too many SSTables.
> >
> > =Rob
> > http://twitter.com/rcolidba
> > PS - Given 30GB of RAM on the machine, you could consider investigating
> "large-heap" configurations, rbranson from Instagram has some slides out
> there on the topic. What you pay is longer stop the world GCs, IOW latency
> if you happen to be talking to a replica node when it pauses.
> >
>

[Import csv to Cassandra] Taking too much time

[严超]

Hi, Everyone:
I'm importing a CSV file into Cassandra, and it always get
error:"Request did not complete within rpc_timeout ", then I have to
continue my COPY command of cql again. And the CSV file is 2.2 G . It is
taking a long time.
How can I speed up csv file importing ? Is there another way except
COPY command of cql ?
Thank you for any help .

*Best Regards！*


*Chao Yan--**My twitter：Andy Yan @yanchao727
*


*My Weibo：http://weibo.com/herewearenow
--*

mirror between github and apache

[Christian Andersson]

Hi,

Have a question regarding the mirror of git://git.apache.org/cassandra.git
and github.

How are the repositories synchronized? Manually, some kind of automatic job
doing this..

//chibbe