Cassandra HEAP Suggestion.. Need a help
Hi Team, I have 64GB of total system memory. 5 node cluster. x ~# free -m totalusedfree shared buff/cache available Mem: 64266 17549 41592 665124 46151 Swap: 0 0 0 x ~# and "egrep -c 'processor([[:space:]]+):.*' /proc/cpuinfo" giving 12 cpu cores. Currently Cassandra-env.sh calculating MAX_HEAP_SIZE as '8GB' and HEAP_NEWSIZE as '1200 MB' I am facing Java insufficient memory issue and Cassandra service is getting down. I going to hard code the HEAP values in Cassandra-env.sh as below. MAX_HEAP_SIZE="16G" (1/4 of total RAM) HEAP_NEWSIZE="4G" (1/4 of MAX_HEAP_SIZE) Is these values correct for my setup in production? Is there any disadvantages doing this? Please let me know if any of you people faced the same issue. Thanks in advance! Best regards, Bhargav M
Commitlogs are filling the Full Disk space and nodes are down
Hi Team, My Cassandra version : Apache Cassandra 3.0.13 Cassandra nodes are down due to Commitlogs are getting filled up until full disk size. [cid:image001.jpg@01D399E3.666CF940] With "Nodetool flush" I didn't see any commitlogs deleted. Can anyone tell me how to flush the commitlogs without losing data. Thanks, Bhargav M
Cassandra nodes are down
Hi Team, I'm getting the below warnings. Please help me out to clear these issues. Apache Cassandra version : 3.0.13, 5 Node cluster INFO [main] 2018-01-29 16:58:19,487 NativeLibrary.java:167 - JNA mlockall successful WARN [main] 2018-01-29 16:58:19,488 StartupChecks.java:121 - jemalloc shared library could not be preloaded to speed up memory allocations INFO [main] 2018-01-29 16:58:19,488 StartupChecks.java:160 - JMX is enabled to receive remote connections on port: 8002 WARN [main] 2018-01-29 16:58:19,488 StartupChecks.java:178 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release INFO [main] 2018-01-29 16:58:19,490 SigarLibrary.java:44 - Initializing SIGAR library WARN [main] 2018-01-29 16:58:19,498 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : true, Address space adequate? : true, nofile limit adequate? : false, nproc limit adequate? : true WARN [main] 2018-01-29 16:58:19,500 StartupChecks.java:246 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl. WARN [main] 2018-01-29 17:05:07,844 SystemKeyspace.java:1042 - No host ID found, created 2dc59352-e98e-4e77-a5f2-289697e467c7 (Note: This should happen exactly once per node). INFO [main] 2018-01-29 17:05:16,421 Server.java:160 - Starting listening for CQL clients on /10.50.21.22:9042 (unencrypted)... INFO [main] 2018-01-29 17:05:16,449 CassandraDaemon.java:488 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it INFO [OptionalTasks:1] 2018-01-29 17:05:18,443 CassandraRoleManager.java:350 - Created default superuser role 'cassandra' INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,737 HintsService.java:212 - Paused hints dispatch INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,740 Server.java:180 - Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,740 Gossiper.java:1490 - Announcing shutdown INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,741 StorageService.java:1991 - Node /10.50.21.22 state jump to shutdown INFO [StorageServiceShutdownHook] 2018-01-29 17:09:57,743 MessagingService.java:811 - Waiting for messaging service to quiesce INFO [ACCEPT-/10.50.21.22] 2018-01-29 17:09:57,743 MessagingService.java:1110 - MessagingService has terminated the accept() thread INFO [StorageServiceShutdownHook] 2018-01-29 17:09:57,797 HintsService.java:212 - Paused hints dispatch Thanks, Bhargav M.
Cassandra Upgrade and Driver compatibility
Hi All, We are planning to upgrade the Apache Cassandra and CPP driver as below. Cassandra service version : Old - cassandra-3.0.6-1.noarch.rpm New - cassandra-3.0.13-1.noarch.rpm Cassandra driver version : Old - cassandra-cpp-driver-2.4.3-1.el7.centos.x86_64.rpm New - cassandra-cpp-driver-2.6.0-1.el7.centos.x86_64.rpm We need a confirmation on below combinations of compatibility. 1. if we upgrade to Cassandra 3.0.13 version and continue with old CPP driver "cassandra-cpp-driver-2.4.3-1.el7.centos.x86_64.rpm" does this combination works? Is there any incompatibility? 1. Vice versa to case1, if we continue with old Cassandra 3.0.6 version and upgraded only CPP driver to "cassandra-cpp-driver-2.6.0-1.el7.centos.x86_64.rpm " does this combination works? Is there any incompatibility? Also, please suggest what is the most stable version of Apache Cassandra later to 3.0.13 ? Thanks in advance.. Best Regards, Bhargav
Securing Cassandra database
Hi Team, We are using Apache Cassandra 3.0.13 version. As part of Cassandra database security, we have created database super user authentication, but from driver side we have default cql connection syntax as "cqlsh " not like "cqlsh -u username and -p password". So cqlsh connection failing from application side. So we have choosen a firewall method to limit the access to Cassandra database with system IP address ranges. Suggest us If any other better method than IP address firewall to create a security for Cassandra. Thanks, Bhargav
issue in creating a role for cassandra database
Hi , I am getting the below issue while creating a role in Cassandra. Single node , Cqlsh is pointing to '127.0.0.1'. [cid:image001.png@01D358C3.8ABF82B0] Can anyone help me in this issue? Thanks, Bhargav
Cassandra issue “Unable to gossip with any seeds”
Hi Team, I am getting the below issue when I started a node. “Unable to gossip with any seeds” [cid:image003.jpg@01D2F4F3.8B790480] I am using Apache Cassandra 3.0.6 version. I have 5 nodes in the cluster,10.50.20.151 , 10.50.20.152 , 10.50.20.153 , 10.50.20.154 , 10.50.20.155 Seed nodes - 10.50.20.153 , 10.50.20.154 , 10.50.20.155 All the nodes have same Cassandra.yaml configuration file I am getting this issue only in node2 (10.50.20.152). It didn’t even joined the cluster. [cid:image004.png@01D2F4F1.9773CC90] Also node2 is pinging from seed nodes and vice versa. [cid:image005.png@01D2F4F2.4B7CAA90] Please let me know if any of you have faced the same issue and found the solution. Thank you very much Regards, Bhargav
Maximum memory usage reached in cassandra!
Hi Cassandra users, I am getting "Maximum memory usage reached (536870912 bytes), cannot allocate chunk of 1048576 bytes" . As a remedy I have changed the off heap memory usage limit cap i.e file_cache_size_in_mb parameter in cassandra.yaml from 512 to 1024. But now again the increased limit got filled up and throwing a message "Maximum memory usage reached (1073741824 bytes), cannot allocate chunk of 1048576 bytes" This issue occurring when redistribution of index's happening ,due to this Cassandra nodes are still UP but read requests are getting failed from application side. My configuration details are as below: 5 node cluster , each node with 68 disks, each disk is 3.7 TB Total CPU cores - 8 total Mem:377G used 265G free 58G shared 378M buff/cache 53G available 104G MAX_HEAP_SIZE is 4GB file_cache_size_in_mb: 1024 memtable heap space is commented in yaml file as below: # memtable_heap_space_in_mb: 2048 # memtable_offheap_space_in_mb: 2048 Can anyone please suggest the solution for this issue. Thanks in advance ! Thanks, Bhargav M
