Re: Secure data

[rajpal reddy]

Hi Jon,

Was trying the LUKS  encryption following the Doc. 
https://aws.amazon.com/blogs/security/how-to-protect-data-at-rest-with-amazon-ec2-instance-store-encryption/
 

 on ec2 i3.large machine.
i don’t see the disk mounted.  and see the mapper being at 100%. do you see 
anything wrong following below statements.
i see this error is /var/log/messages
 ERROR [instanceID=i-0de508d7fc188ab20] [MessagingDeliveryService] 
[Association] Unable to load instance associations, unable to retrieve 
associations unable to retrieve associations NoCredentialProviders: no valid 
providers in chain. Deprecated

df -h /dev/mapper/
Filesystem  Size  Used Avail Use% Mounted on
devtmpfs7.5G  7.5G 0 100% /dev

#!/bin/bash

## Initial setup to be executed on boot
##

# Create an empty file. This file will be used to host the file system.
# In this example we create a 2 GB file called secretfs (Secret File System).
dd of=secretfs bs=1G count=0 seek=2
# Lock down normal access to the file.
chmod 600 secretfs
# Associate a loopback device with the file.
losetup /dev/nvme0 secretfs
#Copy encrypted password file from S3. The password is used to configure LUKE 
later on.
aws s3 cp s3://mybucket/LuksInternalStorageKey .
# Decrypt the password from the file with KMS, save the secret password in 
LuksClearTextKey
LuksClearTextKey=$(aws --region us-east-1 kms decrypt --ciphertext-blob 
fileb://LuksInternalStorageKey --output text --query Plaintext | base64 
--decode)
# Encrypt storage in the device. cryptsetup will use the Linux
# device mapper to create, in this case, /dev/mapper/secretfs.
# Initialize the volume and set an initial key.
echo "$LuksClearTextKey" | cryptsetup -y luksFormat /dev/nvme0
# Open the partition, and create a mapping to /dev/mapper/secretfs.
echo "$LuksClearTextKey" | cryptsetup luksOpen /dev/nvme0 secretfs
# Clear the LuksClearTextKey variable because we don't need it anymore.
unset LuksClearTextKey
# Check its status (optional).
cryptsetup status secretfs
# Zero out the new encrypted device.
dd if=/dev/zero of=/dev/mapper/secretfs
# Create a file system and verify its status.
mke2fs -j -O dir_index /dev/mapper/secretfs
# List file system configuration (optional).
tune2fs -l /dev/mapper/secretfs
# Mount the new file system to /data_e/secretfs.
sudo mkdir /data_e/secretfs
sudo mount /dev/mapper/secretfs /data_e/secretfs


> On Aug 1, 2018, at 3:38 PM, Jonathan Haddad  wrote:
> 
> You can also get full disk encryption with LUKS, which I've used before.
> 
> On Wed, Aug 1, 2018 at 12:36 PM Jeff Jirsa  > wrote:
> EBS encryption worked well on gp2 volumes (never tried it on any others)
> 
> -- 
> Jeff Jirsa
> 
> 
> On Aug 1, 2018, at 7:57 AM, Rahul Reddy  > wrote:
> 
>> Hello,
>> 
>> Any one tried aws ec2 volume encryption for Cassandra instances?
>> 
>> On Tue, Jul 31, 2018, 12:25 PM Rahul Reddy > > wrote:
>> Hello,
>> 
>> I'm trying to find a good document on to enable encryption for Apache 
>> Cassandra  (not on dse) tables and commilogs and store the keystore in kms 
>> or vault. If any of you already configured please direct me to documentation 
>> for it.
> 
> 
> -- 
> Jon Haddad
> http://www.rustyrazorblade.com 
> twitter: rustyrazorblade

Re: Jmx_exporter CPU spike

[rajpal reddy]

Thanks Ben!. will look into it
> On Jul 9, 2018, at 10:42 AM, Ben Bromhead  wrote:
> 
> Hi Rajpal
> 
> I'd invite you to have a look at 
> https://github.com/zegelin/cassandra-exporter 
> <https://github.com/zegelin/cassandra-exporter> 
> 
> Significantly faster (bypasses JMX rpc stuff, 10ms to collect metrics for 300 
> tables vs 2-3 seconds via JMX), plus the naming/tagging fits far better into 
> the Prometheus world. Still missing a few stats like GC etc, but feel free to 
> submit a PR!
> 
> Ben
> 
> 
> 
> On Mon, Jul 9, 2018 at 12:03 AM Rahul Singh  <mailto:rahul.xavier.si...@gmail.com>> wrote:
> How often are you polling the JMX? How much of a spike are you seeing in CPU?
> 
> --
> Rahul Singh
> rahul.si...@anant.us <mailto:rahul.si...@anant.us>
> 
> Anant Corporation
> On Jul 5, 2018, 2:45 PM -0500, rajpal reddy  <mailto:rajpalreddy...@gmail.com>>, wrote:
>> 
>> we have Qualys security scan running causing the cpu spike. We are seeing 
>> the CPU spike only when Jmx metrics are exposed using Jmx_exporter. tried 
>> setting up imx authentication still see cpu spike. if i stop using jmx 
>> exporter we don’t see any cpu spike. is there any thing we have to tune to 
>> make work with Jmx_exporter?
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>> <mailto:user-unsubscr...@cassandra.apache.org>
>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>> <mailto:user-h...@cassandra.apache.org>
>> 
> -- 
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Jmx_exporter CPU spike

[rajpal reddy]

Hi Rahul,

we use prometheus to pull the metrics every 30 secs. cpu is spiking up to 100%. 
when a qualys scan runs net stat shows 100’s connections from the qualys 
server. if i don’t have imx_exporter it doesn’t open soma connections.
> On Jul 9, 2018, at 12:02 AM, Rahul Singh  wrote:
> 
> How often are you polling the JMX? How much of a spike are you seeing in CPU?
> 
> --
> Rahul Singh
> rahul.si...@anant.us
> 
> Anant Corporation
> On Jul 5, 2018, 2:45 PM -0500, rajpal reddy , wrote:
>> 
>> we have Qualys security scan running causing the cpu spike. We are seeing 
>> the CPU spike only when Jmx metrics are exposed using Jmx_exporter. tried 
>> setting up imx authentication still see cpu spike. if i stop using jmx 
>> exporter we don’t see any cpu spike. is there any thing we have to tune to 
>> make work with Jmx_exporter?
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 

Re: Installation

[rajpal reddy]

We have our infrastructure in cloud so opted for adding new dc with tar.gz then 
removed the old dc with package installation

Sent from my iPhone

> On Jul 9, 2018, at 2:23 PM, rajasekhar kommineni  wrote:
> 
> Hello All,
> 
> I have a cassandra cluster where package installation is done, I want to 
> convert it to tar.gz installation. Is there any procedure to follow.
> 
> Thanks,
> Rajasekhar Kommineni
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Jmx_exporter CPU spike

[rajpal reddy]

we have Qualys security scan running causing the cpu spike. We are seeing the 
CPU spike only when Jmx metrics are exposed using Jmx_exporter.  tried setting 
up imx authentication still see cpu spike. if i stop using jmx exporter  we 
don’t see any cpu spike. is there any thing we have to tune to make work with 
Jmx_exporter?


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

CPU Spike with Jmx_exporter

[rajpal reddy]

We are seeing the CPU spike only when Jmx metrics are exposed using 
Jmx_exporter.  tried setting up imx authentication still see cpu spike. if i 
stop using jmx exporter  we don’t see any cpu spike. is there any thing we have 
to tune to make work with Jmx_exporter?


> On Jun 14, 2018, at 2:18 PM, rajpal reddy  wrote:
> 
> Hey Chris,
> 
> Sorry to bother you. Did you get a chance to look at the gclog file I sent 
> last night.
> 
> On Wed, Jun 13, 2018, 8:44 PM rajpal reddy  <mailto:rajpalreddy...@gmail.com>> wrote:
> Chris,
> 
> sorry attached wrong log file. attaching gc collection seconds and cpu. there 
> were going high at the same time and also attached the gc.log. grafana 
> dashboard and gc.log timing are 4hours apart gc can be see 06/12th around 
> 22:50
> 
> rate(jvm_gc_collection_seconds_sum{"}[5m])
> 
> > On Jun 13, 2018, at 5:26 PM, Chris Lohfink  > <mailto:clohf...@apple.com>> wrote:
> > 
> > There are not even a 100ms GC pause in that, are you certain theres a 
> > problem?
> > 
> >> On Jun 13, 2018, at 3:00 PM, rajpal reddy  >> <mailto:rajpalreddy...@gmail.com>> wrote:
> >> 
> >> Thanks Chris I did attached the gc logs already. reattaching them 
> >> now.
> >> 
> >> it started yesterday around 11:54PM 
> >>> On Jun 13, 2018, at 3:56 PM, Chris Lohfink  >>> <mailto:clohf...@apple.com>> wrote:
> >>> 
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>> 
> >>> 
> >>> it depends on the object allocation rate vs the size of the heap. 
> >>> Cassandra ideally would be sub 500-600mb/s allocations but it can spike 
> >>> pretty high with something like reading a wide partition or repair 
> >>> streaming which might exceed what the g1 ygcs tenuring and timing is 
> >>> prepared for from previous steady rate. Giving it a bigger buffer is a 
> >>> nice safety net for allocation spikes.
> >>> 
> >>>> is the HEAP_NEWSIZE is required only for CMS
> >>> 
> >>> 
> >>> it should only set Xmn with that if using CMS, with G1 it should be 
> >>> ignored or else yes it would be bad to set Xmn. Giving the gc logs will 
> >>> give the results of all the bash scripts along with details of whats 
> >>> happening so its your best option if you want help to share that.
> >>> 
> >>> Chris
> >>> 
> >>>> On Jun 13, 2018, at 12:17 PM, Subroto Barua 
> >>>>  wrote:
> >>>> 
> >>>> Chris,
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>> 
> >>>> Subroto 
> >>>> 
> >>>>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink  >>>>> <mailto:clohf...@apple.com>> wrote:
> >>>>> 
> >>>>> G1ReservePercent
> >>>> 
> >>>> -
> >>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> >>>> <mailto:user-unsubscr...@cassandra.apache.org>
> >>>> For additional commands, e-mail: user-h...@cassandra.apache.org 
> >>>> <mailto:user-h...@cassandra.apache.org>
> >>>> 
> >>> 
> >>> 
> >>> -
> >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> >>> <mailto:user-unsubscr...@cassandra.apache.org>
> >>> For additional commands, e-mail: user-h...@cassandra.apache.org 
> >>> <mailto:user-h...@cassandra.apache.org>
> >>> 
> >> 
> >> 
> >> 
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> >> <mailto:user-unsubscr...@cassandra.apache.org>
> >> For additional commands, e-mail: user-h...@cassandra.apache.org 
> >> <mailto:user-h...@cassandra.apache.org>
> > 
> > 
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> > <mailto:user-unsubscr...@cassandra.apache.org>
> > For additional commands, e-mail: user-h...@cassandra.apache.org 
> > <mailto:user-h...@cassandra.apache.org>
> > 
> 

Re: sstableloader from dse 4.8.4 to apache cassandra 3.11.1

[rajpal reddy]

Never mind found it. its not a supported version.

> On Jun 19, 2018, at 2:41 PM, rajpal reddy  wrote:
> 
> 
> Hello,
> 
> I’m trying to use sstablloader from dse 4.8.4( 2.1.12) to apache 3.11.1, i’m 
> getting below error. but works fine when i use stableloader dse 5.1.2(apache 
> 3.11.0)
> Could not retrieve endpoint ranges: 
> java.io.IOException: Failed to open transport to: host-ip:9160.
> 
> Any work around to use the stable loader from use 4.8.4(apache 2.1.12) to 
> apache 3.11.1
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

sstableloader from dse 4.8.4 to apache cassandra 3.11.1

[rajpal reddy]

Hello,

I’m trying to use sstablloader from dse 4.8.4( 2.1.12) to apache 3.11.1, i’m 
getting below error. but works fine when i use stableloader dse 5.1.2(apache 
3.11.0)
Could not retrieve endpoint ranges: 
java.io.IOException: Failed to open transport to: host-ip:9160.

Any work around to use the stable loader from use 4.8.4(apache 2.1.12) to 
apache 3.11.1


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: G1GC CPU Spike

[rajpal reddy]

Hey Chris,

Sorry to bother you. Did you get a chance to look at the gclog file I sent
last night.

On Wed, Jun 13, 2018, 8:44 PM rajpal reddy  wrote:

> Chris,
>
> sorry attached wrong log file. attaching gc collection seconds and cpu.
> there were going high at the same time and also attached the gc.log.
> grafana dashboard and gc.log timing are 4hours apart gc can be see 06/12th
> around 22:50
>
> rate(jvm_gc_collection_seconds_sum{"}[5m])
>
> > On Jun 13, 2018, at 5:26 PM, Chris Lohfink  wrote:
> >
> > There are not even a 100ms GC pause in that, are you certain theres a
> problem?
> >
> >> On Jun 13, 2018, at 3:00 PM, rajpal reddy 
> wrote:
> >>
> >> Thanks Chris I did attached the gc logs already. reattaching them
> now.
> >>
> >> it started yesterday around 11:54PM
> >>> On Jun 13, 2018, at 3:56 PM, Chris Lohfink  wrote:
> >>>
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>
> >>>
> >>> it depends on the object allocation rate vs the size of the heap.
> Cassandra ideally would be sub 500-600mb/s allocations but it can spike
> pretty high with something like reading a wide partition or repair
> streaming which might exceed what the g1 ygcs tenuring and timing is
> prepared for from previous steady rate. Giving it a bigger buffer is a nice
> safety net for allocation spikes.
> >>>
> >>>> is the HEAP_NEWSIZE is required only for CMS
> >>>
> >>>
> >>> it should only set Xmn with that if using CMS, with G1 it should be
> ignored or else yes it would be bad to set Xmn. Giving the gc logs will
> give the results of all the bash scripts along with details of whats
> happening so its your best option if you want help to share that.
> >>>
> >>> Chris
> >>>
> >>>> On Jun 13, 2018, at 12:17 PM, Subroto Barua
>  wrote:
> >>>>
> >>>> Chris,
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>>
> >>>> Subroto
> >>>>
> >>>>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink 
> wrote:
> >>>>>
> >>>>> G1ReservePercent
> >>>>
> >>>> -
> >>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >>>> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>>>
> >>>
> >>>
> >>> -
> >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>>
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
>

Re: G1GC CPU Spike

[rajpal reddy]

does setting HEAP_NEWSIZE=“800M” mean young generation can only use 800M?

> On Jun 13, 2018, at 12:51 PM, Steinmaurer, Thomas 
>  wrote:
> 
> Explicitly setting Xmn with G1 basically results in  <>overriding the target 
> pause-time goal, thus should be avoided.
> http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html 
> <http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html>
>  
> Thomas
>  
>  
> From: rajpal reddy [mailto:rajpalreddy...@gmail.com 
> <mailto:rajpalreddy...@gmail.com>] 
> Sent: Mittwoch, 13. Juni 2018 17:27
> To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Subject: Re: G1GC CPU Spike
>  
> we have this has the Heap settings . is the HEAP_NEWSIZE is required only for 
> CMS. can we get rid of that for G1GC so that it can be used?
> MAX_HEAP_SIZE="8192M"
> HEAP_NEWSIZE=“800M"
>  
> On Jun 13, 2018, at 11:15 AM, Chris Lohfink  <mailto:clohf...@apple.com>> wrote:
>  
> That metric is the total number of seconds spent in GC, it will increase over 
> time with every young gc which is expected. Whats interesting is the rate of 
> growth not the fact that its increasing. If graphing tool has option to graph 
> derivative you should use that instead.
>  
> Chris
> 
> 
> On Jun 13, 2018, at 9:51 AM, rajpal reddy  <mailto:rajpalreddy...@gmail.com>> wrote:
>  
> jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young 
> generation seconds count keep increasing
>  
> 
>  
> On Jun 13, 2018, at 9:52 AM, Chris Lohfink  <mailto:clohf...@apple.com>> wrote:
>  
> The gc log file is best to share when asking for help with tuning. The top of 
> file has all the computed args it ran with and it gives details on what part 
> of the GC is taking time. I would guess the CPU spike is from full GCs which 
> with that small heap of a heap is probably from evacuation failures. 
> Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, 
> along with increasing the amount of heap. 8GB is pretty small for G1, might 
> be better off with CMS.
> 
> Chris
> 
> 
> On Jun 13, 2018, at 8:42 AM, rajpal reddy  <mailto:rajpalreddy...@gmail.com>> wrote:
> 
> Hello,
> 
> we are using G1GC and noticing garbage collection taking a while and during 
> that process we are seeing cpu spiking up to 70-80%. can you please let us 
> know. if we have to tune any parameters for that. attaching the cassandra-env 
> file with jam-options.
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> <mailto:user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> <mailto:user-h...@cassandra.apache.org>
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> <mailto:user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> <mailto:user-h...@cassandra.apache.org>
>  
>  
>  
> The contents of this e-mail are intended for the named addressee only. It 
> contains information that may be confidential. Unless you are the named 
> addressee or an authorized designee, you may not copy or use it, or disclose 
> it to anyone else. If you received it in error please notify us immediately 
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) 
> is a company registered in Linz whose registered office is at 4040 Linz, 
> Austria, Freistädterstraße 313

Re: G1GC CPU Spike

[rajpal reddy]

we have this has the Heap settings . is the HEAP_NEWSIZE is required only for 
CMS. can we get rid of that for G1GC so that it can be used?
MAX_HEAP_SIZE="8192M"
HEAP_NEWSIZE=“800M"

> On Jun 13, 2018, at 11:15 AM, Chris Lohfink  wrote:
> 
> That metric is the total number of seconds spent in GC, it will increase over 
> time with every young gc which is expected. Whats interesting is the rate of 
> growth not the fact that its increasing. If graphing tool has option to graph 
> derivative you should use that instead.
> 
> Chris
> 
>> On Jun 13, 2018, at 9:51 AM, rajpal reddy > <mailto:rajpalreddy...@gmail.com>> wrote:
>> 
>> jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young 
>> generation seconds count keep increasing
>> 
>> 
>> 
>>> On Jun 13, 2018, at 9:52 AM, Chris Lohfink >> <mailto:clohf...@apple.com>> wrote:
>>> 
>>> The gc log file is best to share when asking for help with tuning. The top 
>>> of file has all the computed args it ran with and it gives details on what 
>>> part of the GC is taking time. I would guess the CPU spike is from full GCs 
>>> which with that small heap of a heap is probably from evacuation failures. 
>>> Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, 
>>> along with increasing the amount of heap. 8GB is pretty small for G1, might 
>>> be better off with CMS.
>>> 
>>> Chris
>>> 
>>>> On Jun 13, 2018, at 8:42 AM, rajpal reddy >>> <mailto:rajpalreddy...@gmail.com>> wrote:
>>>> 
>>>> Hello,
>>>> 
>>>> we are using G1GC and noticing garbage collection taking a while and 
>>>> during that process we are seeing cpu spiking up to 70-80%. can you please 
>>>> let us know. if we have to tune any parameters for that. attaching the 
>>>> cassandra-env file with jam-options.
>>>> -
>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>>>> <mailto:user-unsubscr...@cassandra.apache.org>
>>>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>>>> <mailto:user-h...@cassandra.apache.org>
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>>> <mailto:user-unsubscr...@cassandra.apache.org>
>>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>>> <mailto:user-h...@cassandra.apache.org>
>>> 
>> 
> 

Re: G1GC CPU Spike

[rajpal reddy]

Chris,

We have total memory on the machine as 16G so we kept 50% to heap . initial we 
had CMS we had same issues so thought of changing to G1GC we have the same 
problem still
> On Jun 13, 2018, at 9:52 AM, Chris Lohfink  wrote:
> 
> The gc log file is best to share when asking for help with tuning. The top of 
> file has all the computed args it ran with and it gives details on what part 
> of the GC is taking time. I would guess the CPU spike is from full GCs which 
> with that small heap of a heap is probably from evacuation failures. 
> Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, 
> along with increasing the amount of heap. 8GB is pretty small for G1, might 
> be better off with CMS.
> 
> Chris
> 
>> On Jun 13, 2018, at 8:42 AM, rajpal reddy  wrote:
>> 
>> Hello,
>> 
>> we are using G1GC and noticing garbage collection taking a while and during 
>> that process we are seeing cpu spiking up to 70-80%. can you please let us 
>> know. if we have to tune any parameters for that. attaching the 
>> cassandra-env file with jam-options.
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: G1GC CPU Spike

[rajpal reddy]

jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young 
generation seconds count keep increasing



> On Jun 13, 2018, at 9:52 AM, Chris Lohfink  wrote:
> 
> The gc log file is best to share when asking for help with tuning. The top of 
> file has all the computed args it ran with and it gives details on what part 
> of the GC is taking time. I would guess the CPU spike is from full GCs which 
> with that small heap of a heap is probably from evacuation failures. 
> Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, 
> along with increasing the amount of heap. 8GB is pretty small for G1, might 
> be better off with CMS.
> 
> Chris
> 
>> On Jun 13, 2018, at 8:42 AM, rajpal reddy  wrote:
>> 
>> Hello,
>> 
>> we are using G1GC and noticing garbage collection taking a while and during 
>> that process we are seeing cpu spiking up to 70-80%. can you please let us 
>> know. if we have to tune any parameters for that. attaching the 
>> cassandra-env file with jam-options.
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

G1GC CPU Spike

[rajpal reddy]

Hello,

we are using G1GC and noticing garbage collection taking a while and during 
that process we are seeing cpu spiking up to 70-80%. can you please let us 
know. if we have to tune any parameters for that. attaching the cassandra-env 
file with jam-options.calculate_heap_sizes()
{
case "`uname`" in
Linux)
system_memory_in_mb=`free -m | awk '/:/ {print $2;exit}'`
system_cpu_cores=`egrep -c 'processor([[:space:]]+):.*' 
/proc/cpuinfo`
;;
FreeBSD)
system_memory_in_bytes=`sysctl hw.physmem | awk '{print $2}'`
system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
;;
SunOS)
system_memory_in_mb=`prtconf | awk '/Memory size:/ {print $3}'`
system_cpu_cores=`psrinfo | wc -l`
;;
Darwin)
system_memory_in_bytes=`sysctl hw.memsize | awk '{print $2}'`
system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
;;
*)
system_memory_in_mb="2048"
system_cpu_cores="2"
;;
esac

if [ "$system_cpu_cores" -lt "1" ]
then
system_cpu_cores="1"
fi

half_system_memory_in_mb=`expr $system_memory_in_mb / 2`
quarter_system_memory_in_mb=`expr $half_system_memory_in_mb / 2`
if [ "$half_system_memory_in_mb" -gt "1024" ]
then
half_system_memory_in_mb="1024"
fi
if [ "$quarter_system_memory_in_mb" -gt "8192" ]
then
quarter_system_memory_in_mb="8192"
fi
if [ "$half_system_memory_in_mb" -gt "$quarter_system_memory_in_mb" ]
then
max_heap_size_in_mb="$half_system_memory_in_mb"
else
max_heap_size_in_mb="$quarter_system_memory_in_mb"
fi
MAX_HEAP_SIZE="${max_heap_size_in_mb}M"

max_sensible_yg_per_core_in_mb="100"
max_sensible_yg_in_mb=`expr $max_sensible_yg_per_core_in_mb "*" 
$system_cpu_cores`

desired_yg_in_mb=`expr $max_heap_size_in_mb / 4`

if [ "$desired_yg_in_mb" -gt "$max_sensible_yg_in_mb" ]
then
HEAP_NEWSIZE="${max_sensible_yg_in_mb}M"
else
HEAP_NEWSIZE="${desired_yg_in_mb}M"
fi
}

java_ver_output=`"${JAVA:-java}" -version 2>&1`
jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' | awk -F'"' 
'NR==1 {print $2}' | cut -d\- -f1`
JVM_VERSION=${jvmver%_*}

if [ "$JVM_VERSION" \< "1.8" ] ; then
echo "Cassandra 3.0 and later require Java 8u40 or later."
exit 1;
fi

if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" -lt 40 ] ; then
echo "Cassandra 3.0 and later require Java 8u40 or later."
exit 1;
fi

jvm=`echo "$java_ver_output" | grep -A 1 'java version' | awk 'NR==2 {print 
$1}'`
case "$jvm" in
OpenJDK)
JVM_VENDOR=OpenJDK
JVM_ARCH=`echo "$java_ver_output" | awk 'NR==3 {print $2}'`
;;
"Java(TM)")
JVM_VENDOR=Oracle
JVM_ARCH=`echo "$java_ver_output" | awk 'NR==3 {print $3}'`
;;
*)
JVM_VENDOR=other
JVM_ARCH=unknown
;;
esac

JVM_OPTS="$JVM_OPTS -Xloggc:$CASSANDRA_LOG/gc.log"


JVM_OPTS_FILE=$CASSANDRA_CONF/jvm.options
for opt in `grep "^-" $JVM_OPTS_FILE`
do
  JVM_OPTS="$JVM_OPTS $opt"
done

echo $JVM_OPTS | grep -q Xmn
DEFINED_XMN=$?
echo $JVM_OPTS | grep -q Xmx
DEFINED_XMX=$?
echo $JVM_OPTS | grep -q Xms
DEFINED_XMS=$?
echo $JVM_OPTS | grep -q UseConcMarkSweepGC
USING_CMS=$?
echo $JVM_OPTS | grep -q UseG1GC
USING_G1=$?


MAX_HEAP_SIZE="8192M"
HEAP_NEWSIZE="800M"

if [ "x$MAX_HEAP_SIZE" = "x" ] && [ "x$HEAP_NEWSIZE" = "x" -o $USING_G1 -eq 0 
]; then
calculate_heap_sizes
elif [ "x$MAX_HEAP_SIZE" = "x" ] ||  [ "x$HEAP_NEWSIZE" = "x" -a $USING_G1 -ne 
0 ]; then
echo "please set or unset MAX_HEAP_SIZE and HEAP_NEWSIZE in pairs when 
using CMS GC (see cassandra-env.sh)"
exit 1
fi

if [ "x$MALLOC_ARENA_MAX" = "x" ] ; then
export MALLOC_ARENA_MAX=4
fi

if [ $DEFINED_XMX -ne 0 ] && [ $DEFINED_XMS -ne 0 ]; then
 JVM_OPTS="$JVM_OPTS -Xms${MAX_HEAP_SIZE}"
 JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}"
elif [ $DEFINED_XMX -ne 0 ] || [ $DEFINED_XMS -ne 0 ]; then
 echo "Please set or unset -Xmx and -Xms flags in pairs on jvm.options 
file."
 exit 1
fi

if [ $DEFINED_XMN -eq 0 ] && [ $DEFINED_XMX -ne 0 ]; then
echo "Please set or unset -Xmx and -Xmn flags in pairs on jvm.options file."
exit 1
elif [ $DEFINED_XMN -ne 0 ] && [ $USING_CMS -eq 0 ]; then
JVM_OPTS="$JVM_OPTS -Xmn${HEAP_NEWSIZE}"
fi

if [ "$JVM_ARCH" = "64-Bit" ] && [ $USING_CMS -eq 0 ]; then
JVM_OPTS="$JVM_OPTS -XX:+UseCondCardMark"
fi

JVM_OPTS="$JVM_OPTS -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler"

JVM_OPTS="$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.3.0.jar"

if [ "x$CASSANDRA_HEAPDUMP_DIR" != "x" ]; then
JVM_OPTS="$JVM_OPTS 
-XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof"
fi

if [