RE: [EXTERNAL] Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
There is definitely a resource risk to having thousands of open connections to
each node. Some of the drivers have (had?) less than optimal default settings,
like acquiring 50 connections per Cassandra node. This is usually overkill. I
think 5-10/node is much more reasonable. It depends on your app architecture
and cluster node count. If there are lots of small micro-services, maybe they
only need 2 connections per node.
Sean Durity – Staff Systems Engineer, Cassandra
From: Sergio
Sent: Wednesday, October 30, 2019 5:39 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Cassandra 3.11.4 Node the load starts to increase after
few minutes to 40 on 4 CPU machine
Hi Reid,
I don't have anymore this loading problem.
I solved by changing the Cassandra Driver Configuration.
Now my cluster is pretty stable and I don't have machines with crazy CPU Load.
The only thing not urgent but I need to investigate is the number of
ESTABLISHED TCP connections. I see just one node having 7K TCP connections
ESTABLISHED while the others are having around 4-6K connection opened. So the
newest nodes added into the cluster have a higher number of ESTABLISHED TCP
connections.
default['cassandra']['sysctl'] = {
'net.ipv4.tcp_keepalive_time' => 60,
'net.ipv4.tcp_keepalive_probes' => 3,
'net.ipv4.tcp_keepalive_intvl' => 10,
'net.core.rmem_max' => 16777216,
'net.core.wmem_max' => 16777216,
'net.core.rmem_default' => 16777216,
'net.core.wmem_default' => 16777216,
'net.core.optmem_max' => 40960,
'net.ipv4.tcp_rmem' => '4096 87380 16777216',
'net.ipv4.tcp_wmem' => '4096 65536 16777216',
'net.ipv4.ip_local_port_range' => '1 65535',
'net.ipv4.tcp_window_scaling' => 1,
'net.core.netdev_max_backlog' => 2500,
'net.core.somaxconn' => 65000,
'vm.max_map_count' => 1048575,
'vm.swappiness' => 0
}
These are my tweaked value and I used the values recommended from datastax.
Do you have something different?
Best,
Sergio
Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback
mailto:rpinchb...@tripadvisor.com>> ha scritto:
Oh nvm, didn't see the later msg about just posting what your fix was.
R
On 10/30/19, 4:24 PM, "Reid Pinchback"
mailto:rpinchb...@tripadvisor.com>> wrote:
Message from External Sender
Hi Sergio,
Assuming nobody is actually mounting a SYN flood attack, then this sounds
like you're either being hammered with connection requests in very short
periods of time, or your TCP backlog tuning is off. At least, that's where
I'd start looking. If you take that log message and google it (Possible SYN
flooding... Sending cookies") you'll find explanations. Or just googling "TCP
backlog tuning".
R
On 10/30/19, 3:29 PM, "Sergio Bilello"
mailto:lapostadiser...@gmail.com>> wrote:
>
>Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel:
TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies.
Check SNMP counters.
-
To unsubscribe, e-mail:
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail:
user-h...@cassandra.apache.org
The information in this Internet Email is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this Email by
anyone else is unauthorized. If you are not the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be taken in
reliance on it, is prohibited and may be unlawful. When addressed to our
clients any opinions or advice contained in this Email are subject to the terms
and conditions expressed in any applicable governing The Home Depot terms of
business or client engagement letter. The Home Depot disclaims all
responsibility and liability for the accuracy and content of this attachment
and for any damages or losses arising from any inaccuracies, errors, viruses,
e.g., worms, trojan horses, etc., or other items of a destructive nature, which
may be contained in this attachment and shall not be liable for direct,
indirect, consequential or special damages in connection with this e-mail
message or its attachment.
Re: Cassandra 4 alpha/alpha2
OOO but still relevant: Would not it be possible to create an Amazon AMI that has all the OS and JVM settings in the right place and from there each developer can tweak the things that need to be adjusted? Best, Sergio Il giorno gio 31 ott 2019 alle ore 12:56 Abdul Patel ha scritto: > Looks like i am messing up or missing something ..will revisit again > > On Thursday, October 31, 2019, Stefan Miklosovic < > stefan.mikloso...@instaclustr.com> wrote: > >> Hi, >> >> I have tested both alpha and alpha2 and 3.11.5 on Centos 7.7.1908 and >> all went fine (I have some custom images for my own purposes). >> >> Update between alpha and alpha2 was just about mere version bump. >> >> Cheers >> >> On Thu, 31 Oct 2019 at 20:40, Abdul Patel wrote: >> > >> > Hey Everyone >> > >> > Did anyone was successfull to install either alpha or alpha2 version >> for cassandra 4.0? >> > Found 2 issues : >> > 1> cassandra-env.sh: >> > JAVA_VERSION varianle is not defined. >> > Jvm-server.options file is not defined. >> > >> > This is fixable and after adding those , the error for cassandra-env.sh >> errora went away. >> > >> > 2> second and major issue the cassandea binary when i try to start says >> syntax error. >> > >> > /bin/cassandea: line 198:exec: : not found. >> > >> > Anyone has any idea on second issue? >> > >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >>
Re: Cassandra 4 alpha/alpha2
Looks like i am messing up or missing something ..will revisit again On Thursday, October 31, 2019, Stefan Miklosovic < stefan.mikloso...@instaclustr.com> wrote: > Hi, > > I have tested both alpha and alpha2 and 3.11.5 on Centos 7.7.1908 and > all went fine (I have some custom images for my own purposes). > > Update between alpha and alpha2 was just about mere version bump. > > Cheers > > On Thu, 31 Oct 2019 at 20:40, Abdul Patel wrote: > > > > Hey Everyone > > > > Did anyone was successfull to install either alpha or alpha2 version for > cassandra 4.0? > > Found 2 issues : > > 1> cassandra-env.sh: > > JAVA_VERSION varianle is not defined. > > Jvm-server.options file is not defined. > > > > This is fixable and after adding those , the error for cassandra-env.sh > errora went away. > > > > 2> second and major issue the cassandea binary when i try to start says > syntax error. > > > > /bin/cassandea: line 198:exec: : not found. > > > > Anyone has any idea on second issue? > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: Cassandra 4 alpha/alpha2
Hi, I have tested both alpha and alpha2 and 3.11.5 on Centos 7.7.1908 and all went fine (I have some custom images for my own purposes). Update between alpha and alpha2 was just about mere version bump. Cheers On Thu, 31 Oct 2019 at 20:40, Abdul Patel wrote: > > Hey Everyone > > Did anyone was successfull to install either alpha or alpha2 version for > cassandra 4.0? > Found 2 issues : > 1> cassandra-env.sh: > JAVA_VERSION varianle is not defined. > Jvm-server.options file is not defined. > > This is fixable and after adding those , the error for cassandra-env.sh > errora went away. > > 2> second and major issue the cassandea binary when i try to start says > syntax error. > > /bin/cassandea: line 198:exec: : not found. > > Anyone has any idea on second issue? > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Cassandra 4 alpha/alpha2
Centos 7.6 On Thursday, October 31, 2019, Jon Haddad wrote: > What artifact did you use and what OS are you on? > > On Thu, Oct 31, 2019 at 12:40 PM Abdul Patel wrote: > >> Hey Everyone >> >> Did anyone was successfull to install either alpha or alpha2 version for >> cassandra 4.0? >> Found 2 issues : >> 1> cassandra-env.sh: >> JAVA_VERSION varianle is not defined. >> Jvm-server.options file is not defined. >> >> This is fixable and after adding those , the error for cassandra-env.sh >> errora went away. >> >> 2> second and major issue the cassandea binary when i try to start says >> syntax error. >> >> /bin/cassandea: line 198:exec: : not found. >> >> Anyone has any idea on second issue? >> >>
Re: Cassandra 4 alpha/alpha2
What artifact did you use and what OS are you on? On Thu, Oct 31, 2019 at 12:40 PM Abdul Patel wrote: > Hey Everyone > > Did anyone was successfull to install either alpha or alpha2 version for > cassandra 4.0? > Found 2 issues : > 1> cassandra-env.sh: > JAVA_VERSION varianle is not defined. > Jvm-server.options file is not defined. > > This is fixable and after adding those , the error for cassandra-env.sh > errora went away. > > 2> second and major issue the cassandea binary when i try to start says > syntax error. > > /bin/cassandea: line 198:exec: : not found. > > Anyone has any idea on second issue? > >
Cassandra 4 alpha/alpha2
Hey Everyone Did anyone was successfull to install either alpha or alpha2 version for cassandra 4.0? Found 2 issues : 1> cassandra-env.sh: JAVA_VERSION varianle is not defined. Jvm-server.options file is not defined. This is fixable and after adding those , the error for cassandra-env.sh errora went away. 2> second and major issue the cassandea binary when i try to start says syntax error. /bin/cassandea: line 198:exec: : not found. Anyone has any idea on second issue?
what happened to a pagination if some data are inserted before it get resumed ?
Hi, what would happen If between the moment I save a paging state and the moment I resume it, some data have been added to the database ? for example, let say I do a query which return 100 results paged by 10 rows. I get my first page, i.e, my first 10 elements. Then, let say that before I ask to get the second page, some data were added to what was my first page, some to the second etc. what will I see when I resume the pagination ? will I get the results as if nothing was added to the database, or am I going to see on my second page some results that was pushed out of the first page ? in my case we using the python driver our code use the same functions that the following example, class Items(Model): id = columns.Text(primary_key=True) data = columns.Bytes() query = Items.objects.all().limit(10) first_page = list(query); last = first_page[-1] next_page = list(query.filter(pk__token__gt=cqlengine.Token(last.pk))) source: https://docs.datastax.com/en/developer/python-driver/3.20/cqlengine/queryset/ (https://docs.datastax.com/en/developer/python-driver/3.20/cqlengine/queryset/) there is another way to store the pagination than storing the token ? (if I showing the example and asking that it's because I have the feeling there is two ways to use the python driver. the one using function like filter, and another one where we send a query as we would have written in cqlsh) regards, Nicolas Jäger
Re: oversized partition detection ? monitoring the partitions growth ?
Hi, I'm not sure that your are able to log which partition has reached 100MB but you may monitor the "EstimatedPartitionSizeHistogram"and take the max value (or 99ct, 95ct) to trigger an alert using your monitoring system. http://cassandra.apache.org/doc/latest/operating/metrics.html#table-metrics regards, Eric Le 31/10/2019 à 12:37, jagernico...@legtux.org a écrit : Hi, how can I detect a partition that reaches the 100MB ? is it possible to log the size of every partition one time per day ? regards, Nicolas Jäger
oversized partition detection ? monitoring the partitions growth ?
Hi, how can I detect a partition that reaches the 100MB ? is it possible to log the size of every partition one time per day ? regards, Nicolas Jäger
