Nodetool clearsnapshot does not delete snapshot for dropped column_family
Hi guys! I am running cassandra 3.11.4. I dropped a column_family but I am able to see the disk space occupied by that column_family in the disk. I understood that since I have the auto_snapshot flag = true this behavior is expected. However, I would like to avoid to write a dummy script that removes the column_family folder for each node. I tried the nodetool clearsnapshot command but it didn't work and when I try to nodetool listsnapshots I don't see anything. It is like hidden that space occupied. Any suggestion? Thanks, Sergio - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
IN OPERATOR VS BATCH QUERY
Hi guys!
Let's say we have a KEY-VALUE schema
The goal is to delete the KEYS in batches without burning the cluster and be
efficient as soon as possible
I would like to know if it is better to run the query with DELETE FROM
KEY_VALUE_COLUMN_FAMILY WHERE KEY IN ('A','B','C'); At most 10 KEYS in the IN
STATEMENT
OR
HANDLE WITH A CASSANDRA BATCH QUERY and in particular, I was looking at
https://docs.spring.io/spring-data/cassandra/docs/current/api/org/springframework/data/cassandra/core/ReactiveCassandraBatchOperations.html#delete-java.lang.Iterable-
Thanks,
Sergio
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
nodetool load does not match du
Hello! I was trying to understand the below differences: Cassandra 3.11.4 i3xlarge aws nodes $ du -sh /mnt 123G/mnt $ nodetool info ID : 3647fcca-688a-4851-ab15-df36819910f4 Gossip active : true Thrift active : true Native Transport active: true Load : 183.55 GiB Generation No : 1570757970 Uptime (seconds) : 10041867 Heap Memory (MB) : 3574.09 / 7664.00 Off Heap Memory (MB) : 441.70 Data Center: live Rack : us-east-1b Exceptions : 0 Key Cache : entries 1430578, size 100 MiB, capacity 100 MiB, 10075279019 hits, 13328775396 requests, 0.756 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Chunk Cache: entries 7680, size 479.97 MiB, capacity 480 MiB, 1835784783 misses, 11836353728 requests, 0.845 recent hit rate, 141.883 microseconds miss latency Percent Repaired : 0.10752808456509523% Token : (invoke with -T/--tokens to see all 256 tokens) $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 15G 0 15G 0% /dev tmpfs15G 72K 15G 1% /dev/shm tmpfs15G 1.4G 14G 10% /run tmpfs15G 0 15G 0% /sys/fs/cgroup /dev/xvda1 50G 9.9G 41G 20% / /dev/nvme0n1885G 181G 705G 21% /mnt tmpfs 3.0G 0 3.0G 0% /run/user/995 tmpfs 3.0G 0 3.0G 0% /run/user/1009 Why the df -h and du -sh shows a big discrepancy? nodetool load is it computed with df -h? - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
JMX Metrics [Cassandra-Stress-Tool VS JMX]
Hi everyone! Which function has to be used with each JMX Metric Type? https://cassandra.apache.org/doc/latest/operating/metrics.html https://www.datadoghq.com/blog/how-to-collect-cassandra-metrics/ For example: to compute the read latency I did a ratio between ReadTotalLatency_Count JMX Counter and ReadLatency_Count JMX Timer, and the number corresponds to the one exposed via nodetool tablestats . How should I consider the attributes: 95thPercentile, Mean etc... from ReadLatency bean org.apache.cassandra.metrics:type=Table,keyspace=test,scope=test_column_family,name=ReadLatency I found also a grafana open-source dashboard https://grafana.com/grafana/dashboards/5408 but I am not convinced about the displayed numbers if I compare with the numbers shown by the cassandra-stress tool. If I want the QPS does it make sense to use rate(WriteLatencyCount[5m]) in grafana ? The latency computed by the cassandra-stress-tool should almost match the latency shown by the JMX metrics or not? Which one do you monitor ClientRequest metrics or Table metrics or ColumnFamily? I am going to create my Grafana dashboard and explain how I configured it. Best, Sergio - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Find large partition https://github.com/tolbertam/sstable-tools
Thanks! I will look into it On 2019/11/22 19:22:15, Jeff Jirsa wrote: > Brian Gallew has a very simple script that does something similar: > https://github.com/BrianGallew/cassandra_tools/blob/master/poison_pill_tester > > You can also search the logs for messages about writing large partitions > during compaction. > > > > > > On Thu, Nov 21, 2019 at 6:33 PM Sergio Bilello > wrote: > > > Hi guys! > > Just for curiosity do you know anything beside > > https://github.com/tolbertam/sstable-tools to find a large partition? > > Best, > > > > Sergio > > > > - > > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: user-h...@cassandra.apache.org > > > > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Find large partition https://github.com/tolbertam/sstable-tools
Hi guys! Just for curiosity do you know anything beside https://github.com/tolbertam/sstable-tools to find a large partition? Best, Sergio - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
https://docs.datastax.com/en/drivers/java/2.2/com/datastax/driver/core/policies/LatencyAwarePolicy.html I had to change the Policy in the Cassandra Driver. I solved this problem few weeks ago. I am just posting the solution for anyone that could hit the same issue. Best, Sergio On 2019/10/17 02:46:01, Sergio Bilello wrote: > Hello guys! > > I performed a thread dump > https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMTAvMTcvLS1kdW1wLnR4dC0tMC0zMC00MA==; > while try to join the node with > > -Dcassandra.join_ring=false > > OR > -Dcassandra.join.ring=false > > OR > > -Djoin.ring=false > > because the node spiked in load and latency was affecting the clients. > > With or without that flag the node is high in latency and I see the load sky > rocketing when the number of TCP established connections increases > > Analyzing the /var/log/messages I am able to read > > Oct 17 00:23:39 prod-personalization-live-data-cassandra-08 cassandra: INFO > [Service Thread] 2019-10-17 00:23:39,030 GCInspector.java:284 - G1 Young > Generation GC in 255ms. G1 Eden Space: 361758720 -> 0; G1 Old Gen: 1855455944 > -> 1781007048; G1 Survivor Space: 39845888 -> 32505856; > > Oct 17 00:23:40 prod-personalization-live-data-cassandra-08 cassandra: INFO > [ScheduledTasks:1] 2019-10-17 00:23:40,352 NoSpamLogger.java:91 - Some > operations were slow, details available at debug level (debug.log) > > > Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: > request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check > SNMP counters. > > I don't see anything on debug.log that looks to be relevant > > The machine is on aws with 4 cpu with 32GB Ram and 1 TB SSD i3.xlarge > > > > > > [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ nodetool > tpstats > > Pool Name Active Pending Completed Blocked All time blocked > > ReadStage 32 53 559304 0 0 > > MiscStage 0 0 0 0 0 > > CompactionExecutor 1 107 118 0 0 > > MutationStage 0 0 2695 0 0 > > MemtableReclaimMemory 0 0 11 0 0 > > PendingRangeCalculator 0 0 33 0 0 > > GossipStage 0 0 4314 0 0 > > SecondaryIndexManagement 0 0 0 0 0 > > HintsDispatcher 0 0 0 0 0 > > RequestResponseStage 0 0 421865 0 0 > > Native-Transport-Requests 22 0 1903400 0 0 > > ReadRepairStage 0 0 59078 0 0 > > CounterMutationStage 0 0 0 0 0 > > MigrationStage 0 0 0 0 0 > > MemtablePostFlush 0 0 32 0 0 > > PerDiskMemtableFlushWriter_0 0 0 11 0 0 > > ValidationExecutor 0 0 0 0 0 > > Sampler 0 0 0 0 0 > > MemtableFlushWriter 0 0 11 0 0 > > InternalResponseStage 0 0 0 0 0 > > ViewMutationStage 0 0 0 0 0 > > AntiEntropyStage 0 0 0 0 0 > > CacheCleanupExecutor 0 0 0 0 0 > > > > Message type Dropped > > READ 0 > > RANGE_SLICE 0 > > _TRACE 0 > > HINT 0 > > MUTATION 0 > > COUNTER_MUTATION 0 > > BATCH_STORE 0 > > BATCH_REMOVE 0 > > REQUEST_RESPONSE 0 > > PAGED_RANGE 0 > > READ_REPAIR 0 > > [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ > > > > > > top - 01:44:15 up 2 days, 1:45, 4 users, load average: 34.45, 27.71, 15.37 > > Tasks: 140 total, 1 running, 74 sleeping, 0 stopped, 0 zombie > > %Cpu(s): 90.0 us, 4.5 sy, 3.0 ni, 1.1 id, 0.0 wa, 0.0 hi, 1.4 si, 0.0 st > > KiB Mem : 31391772 total, 250504 free, 10880364 used, 20260904 buff/cache > > KiB Swap: 0 total, 0 free, 0 used. 19341960 avail Mem > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 20712 cassand+ 20 0 194.1g 14.4g 4.6g S 392.0 48.2 74:50.48 java > > 20823 sergio.+ 20 0 124856 6304 3136 S 1.7 0.0 0:13.51 htop > > 7865 root 20 0 1062684 39880 11428 S 0.7 0.1 4:06.02 ir_agent > > 3557 consul 20 0 41568 30192 18832 S 0.3 0.1 13:16.37 consul > > 7600 root 20 0 2082700 46624 11880 S 0.3 0.1 4:14.60 ir_agent > > 1 root 20 0 193660 7740 5220 S 0.0 0.0 0:56.36 systemd > > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd > > 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H > > 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq > > 7 root 20 0 0 0 0 S 0.0 0.0 0:06.04 ksoftirqd/0 > > > > > > [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ free > > total used free shared buff/cache available > > Mem: 31391772 10880916 256732 426552 20254124 19341768 > > Swap: 0 0 0 > > [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ > > > > > > > > bash-4.2$ java -jar sjk.jar ttop -p 20712 > > Monitoring threads ... >
Re: Keyspace Clone in Existing Cluster
Rolling bounce = Rolling repair per node? Would not it be easy to be scheduled with Cassandra Reaper? On 2019/10/29 15:35:42, Paul Carlucci wrote: > Copy the schema from your source keyspace to your new target keyspace, > nodetool snapshot on your source keyspace, copy the SSTable files over, do > a rolling bounce, repair, enjoy. In my experience a rolling bounce is > easier than a nodetool refresh. > > It's either that or just copy it with Spark. > > On Tue, Oct 29, 2019, 11:19 AM Ankit Gadhiya wrote: > > > Thanks Alex. So How do I copy SSTables from 1.0 to 2.0? (Same > > SSTableLoader or any other approach?) > > Also since I've multi-node cluster - I'll have to do this on every single > > node - is there any tool or better way to execute this just from a single > > node? > > > > *Thanks & Regards,* > > *Ankit Gadhiya* > > > > > > > > On Tue, Oct 29, 2019 at 11:16 AM Alex Ott wrote: > > > >> You can create all tables in new keyspace, copy SSTables from 1.0 to 2.0 > >> tables & use nodetool refresh on tables in KS 2.0 to say Cassandra about > >> them. > >> > >> On Tue, Oct 29, 2019 at 4:10 PM Ankit Gadhiya > >> wrote: > >> > >>> Hello Folks, > >>> > >>> Greetings!. > >>> > >>> I've a requirement in my project to setup Blue-Green deployment for > >>> Cassandra. E.x. Say My current active schema (application pointing to) is > >>> Keyspace V1.0 and for my next release I want to setup Keysapce 2.0 (with > >>> some structural changes) and all testing/validation would happen on it and > >>> once successful , App would switch connection to keyspace 2.0 - This would > >>> be generic release deployment for our project. > >>> > >>> One of the approach we thought of would be to Create keyspace 2.0 as > >>> clone from Keyspace 1.0 including data using sstableloader but this would > >>> be time consuming, also being a multi-node cluster (6+6 in each DC) - it > >>> wouldn't be very feasible to do this manually on all the nodes for > >>> multiple > >>> tables part of that keyspace. Was wondering if we have any other creative > >>> way to suffice this requirement. > >>> > >>> Appreciate your time on this. > >>> > >>> > >>> *Thanks & Regards,* > >>> *Ankit Gadhiya* > >>> > >>> > >> > >> -- > >> With best wishes,Alex Ott > >> http://alexott.net/ > >> Twitter: alexott_en (English), alexott (Russian) > >> > > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Decommissioned Node UNREACHABLE in describecluster but LEFT in gossipinfo
It disappeared from describecluster after 1 day. It is only in gossipinfo now and this looks to be ok :) On 2019/10/25 04:01:03, Sergio wrote: > Hi guys, > > Cassandra 3.11.4 > > nodetool gossipinfo > /10.1.20.49 > generation:1571694191 > heartbeat:279800 > STATUS:279798:LEFT,-1013739435631815991,1572225050446 > LOAD:279791:3.4105213781E11 > SCHEMA:12:5cad59d2-c3d0-3a12-ad10-7578d225b082 > DC:8:live > RACK:10:us-east-1a > RELEASE_VERSION:4:3.11.4 > INTERNAL_IP:6:10.1.20.49 > RPC_ADDRESS:3:10.1.20.49 > NET_VERSION:1:11 > HOST_ID:2:be5a0193-56e7-4d42-8cc8-5d2141ab4872 > RPC_READY:29:true > TOKENS:15: > > The node is not shown in nodetool status > > and it is displayed as UNREACHABLE in nodetool describecluster > > I found this old conversation > https://grokbase.com/t/cassandra/user/162gwp6pz6/decommissioned-nodes-shows-up-in-nodetool-describecluster-as-unreachable-in-2-1-12-version > > Is there something that I should do to fix this? > > Best, > > Sergio > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Cassandra Rack - Datacenter Load Balancing relations
Hello guys! I was reading about https://cassandra.apache.org/doc/latest/architecture/dynamo.html#networktopologystrategy I would like to understand a concept related to the node load balancing. I know that Jon recommends Vnodes = 4 but right now I found a cluster with vnodes = 256 replication factor = 3 and 2 racks. This is unbalanced because the racks are not a multiplier of the replication factor. However, my plan is to move all the nodes in a single rack to eventually scale up and down the node in the cluster once at the time. If I had 3 racks and I would like to keep the things balanced I should scale up 3 nodes at the time one for each rack. If I would have 3 racks, should I have also 3 different datacenters so one datacenter for each rack? Can I have 2 datacenters and 3 racks? If this is possible one datacenter would have more nodes than the others? Could it be a problem? I am thinking to split my cluster in one datacenter for reads and one for writes and keep all the nodes in the same rack so I can scale up once node at the time. Please correct me if I am wrong Thanks, Sergio - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
Hello! Is it still better to use ParNew + CMS Is it still better than G1GC these days? Any recommendation for i3.xlarge nodes read-heavy workload? Thanks, Sergio - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Cassandra Recommended System Settings
Hello everyone!
Do you have any setting that you would change or tweak from the below list?
sudo cat /proc/4379/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack sizeunlimitedunlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max resident set unlimitedunlimitedbytes
Max processes 3276832768processes
Max open files1048576 1048576 files
Max locked memory unlimitedunlimitedbytes
Max address space unlimitedunlimitedbytes
Max file locksunlimitedunlimitedlocks
Max pending signals unlimitedunlimitedsignals
Max msgqueue size unlimitedunlimitedbytes
Max nice priority 00
Max realtime priority 00
Max realtime timeout unlimitedunlimitedus
These are the sysctl settings
default['cassandra']['sysctl'] = {
'net.ipv4.tcp_keepalive_time' => 60,
'net.ipv4.tcp_keepalive_probes' => 3,
'net.ipv4.tcp_keepalive_intvl' => 10,
'net.core.rmem_max' => 16777216,
'net.core.wmem_max' => 16777216,
'net.core.rmem_default' => 16777216,
'net.core.wmem_default' => 16777216,
'net.core.optmem_max' => 40960,
'net.ipv4.tcp_rmem' => '4096 87380 16777216',
'net.ipv4.tcp_wmem' => '4096 65536 16777216',
'net.ipv4.ip_local_port_range' => '1 65535',
'net.ipv4.tcp_window_scaling' => 1,
'net.core.netdev_max_backlog' => 2500,
'net.core.somaxconn' => 65000,
'vm.max_map_count' => 1048575,
'vm.swappiness' => 0
}
Am I missing something else?
Do you have any experience to configure CENTOS 7
for
JAVA HUGE PAGES
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configRecommendedSettings.html#CheckJavaHugepagessettings
OPTIMIZE SSD
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configRecommendedSettings.html#OptimizeSSDs
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configRecommendedSettings.html
We are using AWS i3.xlarge instances
Thanks,
Sergio
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
Hello guys! I performed a thread dump https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMTAvMTcvLS1kdW1wLnR4dC0tMC0zMC00MA==; while try to join the node with -Dcassandra.join_ring=false OR -Dcassandra.join.ring=false OR -Djoin.ring=false because the node spiked in load and latency was affecting the clients. With or without that flag the node is high in latency and I see the load sky rocketing when the number of TCP established connections increases Analyzing the /var/log/messages I am able to read Oct 17 00:23:39 prod-personalization-live-data-cassandra-08 cassandra: INFO [Service Thread] 2019-10-17 00:23:39,030 GCInspector.java:284 - G1 Young Generation GC in 255ms. G1 Eden Space: 361758720 -> 0; G1 Old Gen: 1855455944 -> 1781007048; G1 Survivor Space: 39845888 -> 32505856; Oct 17 00:23:40 prod-personalization-live-data-cassandra-08 cassandra: INFO [ScheduledTasks:1] 2019-10-17 00:23:40,352 NoSpamLogger.java:91 - Some operations were slow, details available at debug level (debug.log) Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. Check SNMP counters. I don't see anything on debug.log that looks to be relevant The machine is on aws with 4 cpu with 32GB Ram and 1 TB SSD i3.xlarge [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 32 53 559304 0 0 MiscStage 0 0 0 0 0 CompactionExecutor 1 107 118 0 0 MutationStage 0 0 2695 0 0 MemtableReclaimMemory 0 0 11 0 0 PendingRangeCalculator 0 0 33 0 0 GossipStage 0 0 4314 0 0 SecondaryIndexManagement 0 0 0 0 0 HintsDispatcher 0 0 0 0 0 RequestResponseStage 0 0 421865 0 0 Native-Transport-Requests 22 0 1903400 0 0 ReadRepairStage 0 0 59078 0 0 CounterMutationStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemtablePostFlush 0 0 32 0 0 PerDiskMemtableFlushWriter_0 0 0 11 0 0 ValidationExecutor 0 0 0 0 0 Sampler 0 0 0 0 0 MemtableFlushWriter 0 0 11 0 0 InternalResponseStage 0 0 0 0 0 ViewMutationStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 CacheCleanupExecutor 0 0 0 0 0 Message type Dropped READ 0 RANGE_SLICE 0 _TRACE 0 HINT 0 MUTATION 0 COUNTER_MUTATION 0 BATCH_STORE 0 BATCH_REMOVE 0 REQUEST_RESPONSE 0 PAGED_RANGE 0 READ_REPAIR 0 [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ top - 01:44:15 up 2 days, 1:45, 4 users, load average: 34.45, 27.71, 15.37 Tasks: 140 total, 1 running, 74 sleeping, 0 stopped, 0 zombie %Cpu(s): 90.0 us, 4.5 sy, 3.0 ni, 1.1 id, 0.0 wa, 0.0 hi, 1.4 si, 0.0 st KiB Mem : 31391772 total, 250504 free, 10880364 used, 20260904 buff/cache KiB Swap: 0 total, 0 free, 0 used. 19341960 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20712 cassand+ 20 0 194.1g 14.4g 4.6g S 392.0 48.2 74:50.48 java 20823 sergio.+ 20 0 124856 6304 3136 S 1.7 0.0 0:13.51 htop 7865 root 20 0 1062684 39880 11428 S 0.7 0.1 4:06.02 ir_agent 3557 consul 20 0 41568 30192 18832 S 0.3 0.1 13:16.37 consul 7600 root 20 0 2082700 46624 11880 S 0.3 0.1 4:14.60 ir_agent 1 root 20 0 193660 7740 5220 S 0.0 0.0 0:56.36 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq 7 root 20 0 0 0 0 S 0.0 0.0 0:06.04 ksoftirqd/0 [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ free total used free shared buff/cache available Mem: 31391772 10880916 256732 426552 20254124 19341768 Swap: 0 0 0 [sergio.bilello@prod-personalization-live-data-cassandra-08 ~]$ bash-4.2$ java -jar sjk.jar ttop -p 20712 Monitoring threads ... 2019-10-17T01:45:33.352+ Process summary process cpu=363.58% application cpu=261.91% (user=248.65% sys=13.26%) other: cpu=101.67% thread count: 474 heap allocation rate 583mb/s [39] user=13.56% sys=-0.59% alloc= 11mb/s - OptionalTasks:1 [000379] user= 8.57% sys=-0.27% alloc= 18mb/s - ReadStage-19 [000380] user= 7.85% sys= 0.22% alloc= 19mb/s - Native-Transport-Requests-21 [000295] user= 7.14% sys= 0.23% alloc= 14mb/s - Native-Transport-Requests-5 [000378] user= 7.14% sys=-0.03% alloc= 22mb/s - Native-Transport-Requests-17 [000514] user= 6.42% sys= 0.12% alloc= 20mb/s - Native-Transport-Requests-85 [000293] user= 6.66% sys=-0.32% alloc= 12mb/s - Native-Transport-Requests-2 [000392] user= 6.19% sys= 0.14% alloc= 9545kb/s - Native-Transport-Requests-12 [000492] user= 5.71% sys=-0.24% alloc= 15mb/s - Native-Transport-Requests-24 [000294] user= 5.23% sys=-0.25% alloc= 14mb/s - Native-Transport-Requests-3 [000381] user= 5.47% sys=-0.52% alloc= 7430kb/s - Native-Transport-Requests-23 [000672] user= 4.52% sys= 0.25% alloc= 14mb/s - Native-Transport-Requests-270 [000296] user= 5.23% sys=-0.47% alloc= 13mb/s - ReadStage-7 [000673] user= 4.52% sys= 0.05% alloc= 13mb/s - Native-Transport-Requests-269 [000118] user= 4.28% sys=
Cassadra node join problem
Problem: The cassandra node does not work even after restart throwing this exception: WARN [Thread-83069] 2019-10-11 16:13:23,713 CustomTThreadPoolServer.java:125 - Transport error occurred during acceptance of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:109) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:36) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60) ~[libthrift-0.9.2.jar:0.9.2] at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:113) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:134) [apache-cassandra-3.11.4.jar:3.11.4] The CPU Load goes to 50 and it becomes unresponsive. Node configuration: OS: Linux 4.16.13-1.el7.elrepo.x86_64 #1 SMP Wed May 30 14:31:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux This is a working node that does not have the recommended settings but it is working and it is one of the first node in the cluster cat /proc/23935/limits Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimitedseconds Max file size unlimitedunlimitedbytes Max data size unlimitedunlimitedbytes Max stack size8388608 unlimitedbytes Max core file size0unlimitedbytes Max resident set unlimitedunlimitedbytes Max processes 122422 122422 processes Max open files6553665536files Max locked memory 6553665536bytes Max address space unlimitedunlimitedbytes Max file locksunlimitedunlimitedlocks Max pending signals 122422 122422 signals Max msgqueue size 819200 819200 bytes Max nice priority 00 Max realtime priority 00 Max realtime timeout unlimitedunlimitedus I tried to bootstrap a new node that joins the existing cluster. The disk space used is around 400GB SSD over 885GB available At my first attempt, the node failed and got restarted over and over by systemctl that does not honor the limits configuration specified and thrown Caused by: java.nio.file.FileSystemException: /mnt/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-52-big-Index.db: Too many open files at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[na:1.8.0_161] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_161] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:1.8.0_161] at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[na:1.8.0_161] at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_161] at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_161] at org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:104) ~[apache-cassandra-3.11.4.jar:3.11.4] .. 20 common frames omitted ^C I fixed the above by stopping cassandra, cleaning commitlog, saved_caches, hints and data directory and restarting it and getting the PID and run the 2 commands below sudo prlimit -n1048576 -p sudo prlimit -u32768 -p because at the beginning the node didn't even joint the cluster. it was reported by UJ. After fixing the max open file problem, The node from UpJoining passed to the status UpNormal The node joined the cluster but after a while, it started to throw WARN [Thread-83069] 2019-10-11 16:13:23,713 CustomTThreadPoolServer.java:125 - Transport error occurred during acceptance of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:109) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:36) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60) ~[libthrift-0.9.2.jar:0.9.2] at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:113) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:134) [apache-cassandra-3.11.4.jar:3.11.4] I compared
