ccm and loading data

2013-10-30 Thread Kais Ahmed
Hi all,

I'm trying to do some test using ccm (https://github.com/pcmanus/ccm), how
can import data into ? i copied some sstables from production but ccm nod1
refresh do not
exist.

 can anyone tell me which method can i use ?

Thanks,


Re: heap issues - looking for advices on gc tuning

2013-10-30 Thread Jason Tang
What's configuration of following parameters
memtable_flush_queue_size:
concurrent_compactors:


2013/10/30 Piavlo lolitus...@gmail.com

 Hi,

 Below I try to give a full picture to the problem I'm facing.

 This is a 12 node cluster, running on ec2 with m2.xlarge instances (17G
 ram , 2 cpus).
 Cassandra version is 1.0.8
 Cluster normally having between 3000 - 1500 reads per second (depends on
 time of the day) and 1700 - 800 writes per second- according to Opscetner.
 RF=3, now row caches are used.

 Memory relevant  configs from cassandra.yaml:
 flush_largest_memtables_at: 0.85
 reduce_cache_sizes_at: 0.90
 reduce_cache_capacity_to: 0.75
 commitlog_total_space_in_mb: 4096

 relevant JVM options used are:
 -Xms8000M -Xmx8000M -Xmn400M
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
 -XX:MaxTenuringThreshold=1
 -XX:**CMSInitiatingOccupancyFraction**=80 -XX:+**
 UseCMSInitiatingOccupancyOnly

 Now what happens is that with these settings after cassandra process
 restart, the GC it working fine at the beginning, and heap used looks like a
 saw with perfect teeth, eventually the teeth size start to diminish until
 the teeth become not noticable, and then cassandra starts to spend lot's of
 CPU time
 doing gc. It takes about 2 weeks until for such cycle , and then I need to
 restart cassandra process to improve performance.
 During all this time there are no memory  related messages in cassandra
 system.log, except a GC for ParNew: little above 200ms once in a while.

 Things i've already done trying to reduce this eventual heap pressure.
 1) reducing bloom_filter_fp_chance  resulting in reduction from ~700MB to
 ~280MB total per node based on all Filter.db files on the node.
 2) reducing key cache sizes, and dropping key_caches for CFs which do no
 not have many reads
 3) the heap size was increased from 7000M to 8000M
 All these have not really helped , just the increase from 7000M to 8000M,
 helped in increase the cycle till excessive gc from ~9 days to ~14 days.

 I've tried to graph overtime the data that is supposed to be in heap vs
 actual heap size, by summing up all CFs bloom filter sizes + all CFs key
 cache capacities multipled by average key size + all CFs memtables data
 size reported (i've overestimated the data size a bit on purpose to be on
 the safe size).
 Here is a link to graph showing last 2 day metrics for a node which could
 not effectively do GC, and then cassandra process was restarted.
 http://awesomescreenshot.com/**0401w5y534http://awesomescreenshot.com/0401w5y534
 You can clearly see that before and after restart, the size of data that
 is in supposed to be in heap, is the same pretty much the same,
 which makes me think that I really need is GC tunning.

 Also I suppose that this is not due to number of total keys each node has
 , which is between 300 - 200 milions keys for all CF key estimates summed
 on a code.
 The nodes have datasize between 75G to 45G  accordingly to milions of
 keys. And all nodes are starting to have having GC heavy load after about
 14 days.
 Also the excessive GC and heap usage are not affected by load which varies
 depending on time of the day (see read/write rates at the beginning of the
 mail).
 So again based on this , I assume this is not due to large number of keys
 or too much load on the cluster,  but due to a pure GC misconfiguration
 issue.

 Things I remember that I've tried for GC tunning:
 1) Changing -XX:MaxTenuringThreshold=1 to values like 8 - did not help.
 2) Adding  -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:**
 CMSIncrementalDutyCycleMin=0
   -XX:CMSIncrementalDutyCycle=10 -XX:ParallelGCThreads=2
 JVM_OPTS -XX:ParallelCMSThreads=1
 this actually made things worse.
 3) Adding -XX:-XX-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 - did not help.

 Also since it takes like 2 weeks to verify that changing GC setting did
 not help, the process is painfully slow to try all the possibilities :)
 I'd highly appreciate any help and hints on the GC tunning.

 tnx
 Alex









RE: OpsCenter not connecting to Cluster

2013-10-30 Thread Nigel LEACH
Hello Pieter, Thank you for the response. I picked up on the stackoverlow link, 
but I am already setting rpc_server_type as sync. As you say they are Ubuntu, 
and I am Redhat, but I'm not on top RPC implementations to know whether that is 
relevent. 

Another point, in my original mail I missed of two lines from my log

DEBUG: Trying to connect to node xx.xx.xx.xx over thrift
DEBUG: Not returning SASL credentials for xx.xx.xx.xx

Lastly, I can connect successfully to all the JMX's (via VisualVM) on the 
default 7199 port.

Regards, Nigel

-Original Message-
From: pieter.callewa...@be-mobile.be [mailto:pieter.callewa...@be-mobile.be] 
Sent: 29 October 2013 18:30
To: user@cassandra.apache.org
Subject: RE: OpsCenter not connecting to Cluster

Hi Nigel,

I've currently have a simular problem. However, it has only been reproduced on 
Ubuntu...
Are you using hsha as rpc_server_type?

http://stackoverflow.com/questions/19633980/adding-cluster-error-creating-cluster-call-to-cluster-configs-timed-out
 is a guy with the same problem, showing how to reproduce.
I know the ppl of datastax are now investigating this, but no fix yet...

Kind regards,
Pieter Callewaert

-Original Message-
From: Nigel LEACH [mailto:nigel.le...@uk.bnpparibas.com] 
Sent: dinsdag 29 oktober 2013 18:24
To: user@cassandra.apache.org
Subject: OpsCenter not connecting to Cluster

Cassandra 2.0.1
OpsCenter 3.2.2
Java 1.7.0_25
RHEL 6.4

This is a new test cluster with just three nodes, two seed nodes, SSL turned 
off, and GossipingPropertyFileSnitch. Pretty much out of the box environment, 
with both Cassandra and OpsCenter via DataStax yum repository.

Cassandra seems fine, and OpsCenter is installed on a seed node. The OpsCenter 
gui comes up, but is unable to connect to the cluster, I get this error

INFO: Starting factory 
opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 
0x2440290
INFO: twisted.internet.tcp.Connector instance at 0x24402d8 will retry in 2 
seconds
DEBUG: Problem while pinging node: Traceback (most recent call last):
  File /usr/lib/python2.6/site-packages/opscenterd/ThriftService.py, 
line 157, in checkThriftConnection
UserError: User aborted connection: Shutdown requested.
WARN: ProcessingError while calling CreateClusterConfController: Unable to 
connect to cluster
INFO: Stopping factory 
opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 
0x2440290 

Not getting very far troubleshooting it, any clues would be much appreciated. 
Should I try installing the OpsCenter agent manually maybe?

Many Thanks, Nigel





___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.


___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.



RE: OpsCenter not connecting to Cluster

2013-10-30 Thread Nigel LEACH
Fixed - Sort of.

I noticed in the Cassandra system log that thrift was linked to localhost, not 
xx.xx.xx.xx, or its hostname.

Binding thrift service to localhost/127.0.0.1:9160.

So, in the OpsCenter GUI I entererd localhost as the name of a server in the 
Cluster, and all works.

Regards
Nigel

-Original Message-
From: Nigel LEACH 
Sent: 30 October 2013 10:50
To: user@cassandra.apache.org
Subject: RE: OpsCenter not connecting to Cluster

Hello Pieter, Thank you for the response. I picked up on the stackoverlow link, 
but I am already setting rpc_server_type as sync. As you say they are Ubuntu, 
and I am Redhat, but I'm not on top RPC implementations to know whether that is 
relevent. 

Another point, in my original mail I missed of two lines from my log

DEBUG: Trying to connect to node xx.xx.xx.xx over thrift
DEBUG: Not returning SASL credentials for xx.xx.xx.xx

Lastly, I can connect successfully to all the JMX's (via VisualVM) on the 
default 7199 port.

Regards, Nigel

-Original Message-
From: pieter.callewa...@be-mobile.be [mailto:pieter.callewa...@be-mobile.be] 
Sent: 29 October 2013 18:30
To: user@cassandra.apache.org
Subject: RE: OpsCenter not connecting to Cluster

Hi Nigel,

I've currently have a simular problem. However, it has only been reproduced on 
Ubuntu...
Are you using hsha as rpc_server_type?

http://stackoverflow.com/questions/19633980/adding-cluster-error-creating-cluster-call-to-cluster-configs-timed-out
 is a guy with the same problem, showing how to reproduce.
I know the ppl of datastax are now investigating this, but no fix yet...

Kind regards,
Pieter Callewaert

-Original Message-
From: Nigel LEACH [mailto:nigel.le...@uk.bnpparibas.com] 
Sent: dinsdag 29 oktober 2013 18:24
To: user@cassandra.apache.org
Subject: OpsCenter not connecting to Cluster

Cassandra 2.0.1
OpsCenter 3.2.2
Java 1.7.0_25
RHEL 6.4

This is a new test cluster with just three nodes, two seed nodes, SSL turned 
off, and GossipingPropertyFileSnitch. Pretty much out of the box environment, 
with both Cassandra and OpsCenter via DataStax yum repository.

Cassandra seems fine, and OpsCenter is installed on a seed node. The OpsCenter 
gui comes up, but is unable to connect to the cluster, I get this error

INFO: Starting factory 
opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 
0x2440290
INFO: twisted.internet.tcp.Connector instance at 0x24402d8 will retry in 2 
seconds
DEBUG: Problem while pinging node: Traceback (most recent call last):
  File /usr/lib/python2.6/site-packages/opscenterd/ThriftService.py, 
line 157, in checkThriftConnection
UserError: User aborted connection: Shutdown requested.
WARN: ProcessingError while calling CreateClusterConfController: Unable to 
connect to cluster
INFO: Stopping factory 
opscenterd.ThriftService.NoReconnectCassandraClientFactory instance at 
0x2440290 

Not getting very far troubleshooting it, any clues would be much appreciated. 
Should I try installing the OpsCenter agent manually maybe?

Many Thanks, Nigel





___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.


___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.


___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.



Re: heap issues - looking for advices on gc tuning

2013-10-30 Thread srmore
We ran into similar heap issues a while ago for 1.0.11, I am not sure
whether you are at the luxury of upgrading to at-least 1.2.9, we were not.
After a lot of various painful attempts and weeks of testing (just as in
your case) the following settings worked (did not completely relieve the
heap pressure but helped a lot). We still see some heap issues but at-least
it is a bit stable. Unlike in your case we had very heavy reads and writes.
But its good to know that this happens for light load, I was thinking this
was a symptom of heavy load.

-XX:NewSize=1200M
-XX:SurvivorRatio=4
-XX:MaxTenuringThreshold=2


Not sure whether this will help you or not but I think its worth a try.

-sandeep


On Wed, Oct 30, 2013 at 4:34 AM, Jason Tang ares.t...@gmail.com wrote:

 What's configuration of following parameters
 memtable_flush_queue_size:
 concurrent_compactors:


 2013/10/30 Piavlo lolitus...@gmail.com

 Hi,

 Below I try to give a full picture to the problem I'm facing.

 This is a 12 node cluster, running on ec2 with m2.xlarge instances (17G
 ram , 2 cpus).
 Cassandra version is 1.0.8
 Cluster normally having between 3000 - 1500 reads per second (depends on
 time of the day) and 1700 - 800 writes per second- according to Opscetner.
 RF=3, now row caches are used.

 Memory relevant  configs from cassandra.yaml:
 flush_largest_memtables_at: 0.85
 reduce_cache_sizes_at: 0.90
 reduce_cache_capacity_to: 0.75
 commitlog_total_space_in_mb: 4096

 relevant JVM options used are:
 -Xms8000M -Xmx8000M -Xmn400M
 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
 -XX:MaxTenuringThreshold=1
 -XX:**CMSInitiatingOccupancyFraction**=80 -XX:+**
 UseCMSInitiatingOccupancyOnly

 Now what happens is that with these settings after cassandra process
 restart, the GC it working fine at the beginning, and heap used looks like a
 saw with perfect teeth, eventually the teeth size start to diminish until
 the teeth become not noticable, and then cassandra starts to spend lot's of
 CPU time
 doing gc. It takes about 2 weeks until for such cycle , and then I need
 to restart cassandra process to improve performance.
 During all this time there are no memory  related messages in cassandra
 system.log, except a GC for ParNew: little above 200ms once in a while.

 Things i've already done trying to reduce this eventual heap pressure.
 1) reducing bloom_filter_fp_chance  resulting in reduction from ~700MB to
 ~280MB total per node based on all Filter.db files on the node.
 2) reducing key cache sizes, and dropping key_caches for CFs which do no
 not have many reads
 3) the heap size was increased from 7000M to 8000M
 All these have not really helped , just the increase from 7000M to 8000M,
 helped in increase the cycle till excessive gc from ~9 days to ~14 days.

 I've tried to graph overtime the data that is supposed to be in heap vs
 actual heap size, by summing up all CFs bloom filter sizes + all CFs key
 cache capacities multipled by average key size + all CFs memtables data
 size reported (i've overestimated the data size a bit on purpose to be on
 the safe size).
 Here is a link to graph showing last 2 day metrics for a node which could
 not effectively do GC, and then cassandra process was restarted.
 http://awesomescreenshot.com/**0401w5y534http://awesomescreenshot.com/0401w5y534
 You can clearly see that before and after restart, the size of data that
 is in supposed to be in heap, is the same pretty much the same,
 which makes me think that I really need is GC tunning.

 Also I suppose that this is not due to number of total keys each node has
 , which is between 300 - 200 milions keys for all CF key estimates summed
 on a code.
 The nodes have datasize between 75G to 45G  accordingly to milions of
 keys. And all nodes are starting to have having GC heavy load after about
 14 days.
 Also the excessive GC and heap usage are not affected by load which
 varies depending on time of the day (see read/write rates at the beginning
 of the mail).
 So again based on this , I assume this is not due to large number of keys
 or too much load on the cluster,  but due to a pure GC misconfiguration
 issue.

 Things I remember that I've tried for GC tunning:
 1) Changing -XX:MaxTenuringThreshold=1 to values like 8 - did not help.
 2) Adding  -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:**
 CMSIncrementalDutyCycleMin=0
   -XX:CMSIncrementalDutyCycle=10 -XX:ParallelGCThreads=2
 JVM_OPTS -XX:ParallelCMSThreads=1
 this actually made things worse.
 3) Adding -XX:-XX-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 - did not
 help.

 Also since it takes like 2 weeks to verify that changing GC setting did
 not help, the process is painfully slow to try all the possibilities :)
 I'd highly appreciate any help and hints on the GC tunning.

 tnx
 Alex










Re: heap issues - looking for advices on gc tuning

2013-10-30 Thread Piavlo

On 10/30/2013 11:34 AM, Jason Tang wrote:

What's configuration of following parameters
memtable_flush_queue_size:
concurrent_compactors:

memtable_flush_queue_size:  4
concurrent_compactors: 1

Lokking at the metrics I did not see FlushWriter pending issues
as well the compactions are keeping up with the pace.

tnx



2013/10/30 Piavlo lolitus...@gmail.com mailto:lolitus...@gmail.com

Hi,

Below I try to give a full picture to the problem I'm facing.

This is a 12 node cluster, running on ec2 with m2.xlarge instances
(17G ram , 2 cpus).
Cassandra version is 1.0.8
Cluster normally having between 3000 - 1500 reads per second
(depends on time of the day) and 1700 - 800 writes per second-
according to Opscetner.
RF=3, now row caches are used.

Memory relevant  configs from cassandra.yaml:
flush_largest_memtables_at: 0.85
reduce_cache_sizes_at: 0.90
reduce_cache_capacity_to: 0.75
commitlog_total_space_in_mb: 4096

relevant JVM options used are:
-Xms8000M -Xmx8000M -Xmn400M
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=80
-XX:+UseCMSInitiatingOccupancyOnly

Now what happens is that with these settings after cassandra
process restart, the GC it working fine at the beginning, and heap
used looks like a
saw with perfect teeth, eventually the teeth size start to
diminish until the teeth become not noticable, and then cassandra
starts to spend lot's of CPU time
doing gc. It takes about 2 weeks until for such cycle , and then I
need to restart cassandra process to improve performance.
During all this time there are no memory  related messages in
cassandra system.log, except a GC for ParNew: little above 200ms
once in a while.

Things i've already done trying to reduce this eventual heap pressure.
1) reducing bloom_filter_fp_chance  resulting in reduction from
~700MB to ~280MB total per node based on all Filter.db files on
the node.
2) reducing key cache sizes, and dropping key_caches for CFs which
do no not have many reads
3) the heap size was increased from 7000M to 8000M
All these have not really helped , just the increase from 7000M to
8000M, helped in increase the cycle till excessive gc from ~9 days
to ~14 days.

I've tried to graph overtime the data that is supposed to be in
heap vs actual heap size, by summing up all CFs bloom filter sizes
+ all CFs key cache capacities multipled by average key size + all
CFs memtables data size reported (i've overestimated the data size
a bit on purpose to be on the safe size).
Here is a link to graph showing last 2 day metrics for a node
which could not effectively do GC, and then cassandra process was
restarted.
http://awesomescreenshot.com/0401w5y534
You can clearly see that before and after restart, the size of
data that is in supposed to be in heap, is the same pretty much
the same,
which makes me think that I really need is GC tunning.

Also I suppose that this is not due to number of total keys each
node has , which is between 300 - 200 milions keys for all CF key
estimates summed on a code.
The nodes have datasize between 75G to 45G  accordingly to milions
of keys. And all nodes are starting to have having GC heavy load
after about 14 days.
Also the excessive GC and heap usage are not affected by load
which varies depending on time of the day (see read/write rates at
the beginning of the mail).
So again based on this , I assume this is not due to large number
of keys or too much load on the cluster,  but due to a pure GC
misconfiguration issue.

Things I remember that I've tried for GC tunning:
1) Changing -XX:MaxTenuringThreshold=1 to values like 8 - did not
help.
2) Adding  -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing
-XX:CMSIncrementalDutyCycleMin=0
  -XX:CMSIncrementalDutyCycle=10
-XX:ParallelGCThreads=2 JVM_OPTS -XX:ParallelCMSThreads=1
this actually made things worse.
3) Adding -XX:-XX-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 - did
not help.

Also since it takes like 2 weeks to verify that changing GC
setting did not help, the process is painfully slow to try all the
possibilities :)
I'd highly appreciate any help and hints on the GC tunning.

tnx
Alex











Re: heap issues - looking for advices on gc tuning

2013-10-30 Thread Piavlo

On 10/30/2013 03:21 PM, srmore wrote:
We ran into similar heap issues a while ago for 1.0.11, I am not sure 
whether you are at the luxury of upgrading to at-least 1.2.9, we were 
not. After a lot of various painful attempts and weeks of testing 
(just as in your case) the following settings worked (did not 
completely relieve the heap pressure but helped a lot). We still see 
some heap issues but at-least it is a bit stable. Unlike in your case 
we had very heavy reads and writes.
Well that's what I can have with the current 12 node ec2 m2.xlarge 
instances cluster, if reads/writes increases significantly higher during 
peak times , the node would be locked by 100% user + system cpu usage, 
irrelevant of the GC.

If you are on ec2 also what cluster spec and read/write rates do you have?
But its good to know that this happens for light load, I was thinking 
this was a symptom of heavy load.


-XX:NewSize=1200M
wow 1200M is really huge size for NewSize , this will blow up the GC for 
ParNew pause skyes high for me, I remember back then I increased it from 
300M  to 400M i already spotted in crease in ParNew intervals.
are such high NewSize really recommended? I guess it partially also 
depends on hardware specs. Do you record the NewSize GC paues intervals 
with the 1200M setting?

-XX:SurvivorRatio=4
to my knowledge this settings does not have any effect without 
-XX:-XX-UseAdaptiveSizePolicy



-XX:MaxTenuringThreshold=2

i had this changed from 1 to 8 and with 8 it was worse, do you think 2 
could make any better difference?


tnx for your help
Alex


Not sure whether this will help you or not but I think its worth a try.

-sandeep


On Wed, Oct 30, 2013 at 4:34 AM, Jason Tang ares.t...@gmail.com 
mailto:ares.t...@gmail.com wrote:


What's configuration of following parameters
memtable_flush_queue_size:
concurrent_compactors:


2013/10/30 Piavlo lolitus...@gmail.com mailto:lolitus...@gmail.com

Hi,

Below I try to give a full picture to the problem I'm facing.

This is a 12 node cluster, running on ec2 with m2.xlarge
instances (17G ram , 2 cpus).
Cassandra version is 1.0.8
Cluster normally having between 3000 - 1500 reads per second
(depends on time of the day) and 1700 - 800 writes per second-
according to Opscetner.
RF=3, now row caches are used.

Memory relevant  configs from cassandra.yaml:
flush_largest_memtables_at: 0.85
reduce_cache_sizes_at: 0.90
reduce_cache_capacity_to: 0.75
commitlog_total_space_in_mb: 4096

relevant JVM options used are:
-Xms8000M -Xmx8000M -Xmn400M
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=80
-XX:+UseCMSInitiatingOccupancyOnly

Now what happens is that with these settings after cassandra
process restart, the GC it working fine at the beginning, and
heap used looks like a
saw with perfect teeth, eventually the teeth size start to
diminish until the teeth become not noticable, and then
cassandra starts to spend lot's of CPU time
doing gc. It takes about 2 weeks until for such cycle , and
then I need to restart cassandra process to improve performance.
During all this time there are no memory  related messages in
cassandra system.log, except a GC for ParNew: little above
200ms once in a while.

Things i've already done trying to reduce this eventual heap
pressure.
1) reducing bloom_filter_fp_chance  resulting in reduction
from ~700MB to ~280MB total per node based on all Filter.db
files on the node.
2) reducing key cache sizes, and dropping key_caches for CFs
which do no not have many reads
3) the heap size was increased from 7000M to 8000M
All these have not really helped , just the increase from
7000M to 8000M, helped in increase the cycle till excessive gc
from ~9 days to ~14 days.

I've tried to graph overtime the data that is supposed to be
in heap vs actual heap size, by summing up all CFs bloom
filter sizes + all CFs key cache capacities multipled by
average key size + all CFs memtables data size reported (i've
overestimated the data size a bit on purpose to be on the safe
size).
Here is a link to graph showing last 2 day metrics for a node
which could not effectively do GC, and then cassandra process
was restarted.
http://awesomescreenshot.com/0401w5y534
You can clearly see that before and after restart, the size of
data that is in supposed to be in heap, is the same pretty
much the same,
which makes me think that I really need is GC tunning.

Also I suppose that this is not due to number of total keys
each 

Re: ReadCount change rate is different across nodes

2013-10-30 Thread Piavlo

On 10/30/2013 02:06 AM, Daning Wang wrote:
We are running 1.2.5 on 8 nodes(256 tokens). all the nodes are running 
on same type of machine. and db size is about same. but recently we 
checked ReadCount stats through jmx, and found that some nodes got  3 
times change rate(we have calculated the changes per minute)  than 
others.


We are using hector on client side, and clients are connecting to all 
the servers, we checked open connections on each server, the numbers 
are about same.


What could cause this problem, how to debug this?
check per node reads latency CF metrics, and i guess you have dynamic 
snitch enabled?

http://www.datastax.com/dev/blog/dynamic-snitching-in-cassandra-past-present-and-future



Thanks in advance,

Daning





unsubscribe

2013-10-30 Thread Leonid Ilyevsky
Unsubscribe

This email, along with any attachments, is confidential and may be legally 
privileged or otherwise protected from disclosure. Any unauthorized 
dissemination, copying or use of the contents of this email is strictly 
prohibited and may be in violation of law. If you are not the intended 
recipient, any disclosure, copying, forwarding or distribution of this email is 
strictly prohibited and this email and any attachments should be deleted 
immediately.  This email and any attachments do not constitute an offer to sell 
or a solicitation of an offer to purchase any interest in any investment 
vehicle sponsored by Moon Capital Management LP (Moon Capital). Moon Capital 
does not provide legal, accounting or tax advice. Any statement regarding 
legal, accounting or tax matters was not intended or written to be relied upon 
by any person as advice. Moon Capital does not waive confidentiality or 
privilege as a result of this email.


Re: unsubscribe

2013-10-30 Thread Dave Brosius

Please send that same riveting text to user-unsubscr...@cassandra.apache.org


*http://tinyurl.com/kdrwyrc*


On 10/30/2013 02:49 PM, Leonid Ilyevsky wrote:

Unsubscribe

This email, along with any attachments, is confidential and may be legally privileged or 
otherwise protected from disclosure. Any unauthorized dissemination, copying or use of 
the contents of this email is strictly prohibited and may be in violation of law. If you 
are not the intended recipient, any disclosure, copying, forwarding or distribution of 
this email is strictly prohibited and this email and any attachments should be deleted 
immediately.  This email and any attachments do not constitute an offer to sell or a 
solicitation of an offer to purchase any interest in any investment vehicle sponsored by 
Moon Capital Management LP (Moon Capital). Moon Capital does not provide 
legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters 
was not intended or written to be relied upon by any person as advice. Moon Capital does 
not waive confidentiality or privilege as a result of this email.





Re: IllegalStateException when bootstrapping new nodes

2013-10-30 Thread Cyril Scetbon
FYI, we should upgrade to the last 1.2 version (1.2.11+) in January 2014. 
However, we would like to know if it's a known fixed bug or inform you about 
this issue if it's not.
-- 
Cyril SCETBON

On 29 Oct 2013, at 19:14, Robert Coli rc...@eventbrite.com wrote:

 On Tue, Oct 29, 2013 at 7:38 AM, Cyril Scetbon cyril.scet...@free.fr wrote:
 We didn't find the reason why it didn't work but we are wondering if it's a 
 BUG. We are using Cassandra 1.2.2 and we resolved the issue with a rolling 
 restart of other nodes in the same DC and maybe with some luck... We don't 
 know if it's important but we also added the first 2 new nodes added to the 
 topology file before we successfully bootstrapped the last 2 nodes. 
 
 While this is not directly responsive to your question, I suggest upgrading 
 from 1.2.2 to at least 1.2.9 ASAP. 1.2.2 contains significant bugs.
 
 =Rob
  



Re: IllegalStateException when bootstrapping new nodes

2013-10-30 Thread Robert Coli
On Wed, Oct 30, 2013 at 12:22 PM, Cyril Scetbon cyril.scet...@free.frwrote:

 FYI, we should upgrade to the last 1.2 version (1.2.11+) in January 2014.
 However, we would like to know if it's a known fixed bug or inform you
 about this issue if it's not.


Did you bootstrap multiple nodes into the same token range? That is
unsupported...

What does nodetool ring say?

=Rob


need help with triggers

2013-10-30 Thread kaveh minooie

Hi everyone

I am trying to write a trigger and I am having a hard time figuring out 
how to extract data from the name field of the columns. I can read the 
value reletavily easy by getting the abstractype for that column from 
CFMetadata and use it to decode the value bytebuffer, but due to the 
composite nature of the name filed ( in cql columns ) I can't do the 
same thing for name. so to sum up, I need to be able to skip the 
composite key and get the name of the of the column, and also get any 
data that are set in the name field (e.x. columns of type set). any hint 
or suggestion would be greatlly appriciated.


thanks,

p.s. this for casandra 2.0


--
Kaveh Minooie


Re: ReadCount change rate is different across nodes

2013-10-30 Thread Daning Wang
Thanks. actually I forgot to mention it is multi-center environment and we
have dynamic snitch disabled. because we saw some performance impact on the
multi-center environment.





On Wed, Oct 30, 2013 at 11:12 AM, Piavlo lolitus...@gmail.com wrote:

 On 10/30/2013 02:06 AM, Daning Wang wrote:

 We are running 1.2.5 on 8 nodes(256 tokens). all the nodes are running on
 same type of machine. and db size is about same. but recently we checked
 ReadCount stats through jmx, and found that some nodes got  3 times change
 rate(we have calculated the changes per minute)  than others.

 We are using hector on client side, and clients are connecting to all the
 servers, we checked open connections on each server, the numbers are about
 same.

 What could cause this problem, how to debug this?

 check per node reads latency CF metrics, and i guess you have dynamic
 snitch enabled?
 http://www.datastax.com/dev/**blog/dynamic-snitching-in-**
 cassandra-past-present-and-**futurehttp://www.datastax.com/dev/blog/dynamic-snitching-in-cassandra-past-present-and-future



 Thanks in advance,

 Daning