We tested in c4 instances but EBS is too slow. So we deployed for production in i3.
It was running with 5 nodes without problems but we started running out of space so we added another node. And is this last node that is giving problems. I've already terminated the instance and created another once, but the problem remains. The configuration is the same for all instances. Runs nice for a few time but then ERROR [MemtablePostFlush:11] 2017-04-06 11:47:40,840 CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:11,5,main] Thanks. On 04/06/2017 01:18 PM, Carlos Rolo wrote: > There was some issue with the i3 instances and Cassandra. Did you had > this cluster running always on i3? > > On Apr 6, 2017 13:06, "Cogumelos Maravilha" > <cogumelosmaravi...@sapo.pt <mailto:cogumelosmaravi...@sapo.pt>> wrote: > > Limit Soft Limit Hard > Limit Units > Max cpu time unlimited > unlimited seconds > Max file size unlimited > unlimited bytes > Max data size unlimited > unlimited bytes > Max stack size 8388608 > unlimited bytes > Max core file size 0 > unlimited bytes > Max resident set unlimited > unlimited bytes > Max processes 122575 > 122575 processes > Max open files 100000 > 100000 files > Max locked memory unlimited > unlimited bytes > Max address space unlimited > unlimited bytes > Max file locks unlimited > unlimited locks > Max pending signals 122575 > 122575 signals > Max msgqueue size 819200 > 819200 bytes > Max nice priority 0 0 > Max realtime priority 0 0 > Max realtime timeout unlimited > unlimited us > > Please find something wrong there! > > Thanks. > > On 04/06/2017 11:50 AM, benjamin roth wrote: >> Limits: You should check them in /proc/$pid/limits >> >> 2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha >> <cogumelosmaravi...@sapo.pt <mailto:cogumelosmaravi...@sapo.pt>>: >> >> Yes C* is running as cassandra: >> >> cassand+ 2267 1 99 10:18 ? 00:02:56 java >> -Xloggc:/var/log/cassandra/gc.log -ea >> -XX:+UseThreadPriorities -XX:Threa... >> >> INFO [main] 2017-04-06 10:35:42,956 Config.java:474 - Node >> configuration:[allocate_tokens_for_keyspace=null; >> authenticator=PasswordAuthenticator; >> authorizer=CassandraAuthorizer; auto_bootstrap=true; >> auto_snapshot=true; back_pressure_enabled=false; >> back_pressure_strategy=org.apache.cassandra.net >> <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9, >> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; >> batch_size_warn_threshold_in_kb=5; >> batchlog_replay_throttle_in_kb=1024; broadcast_address=null; >> broadcast_rpc_address=null; >> buffer_pool_use_heap_if_exhausted=true; >> cas_contention_timeout_in_ms=6000000; cdc_enabled=false; >> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; >> cdc_total_space_in_mb=0; >> client_encryption_options=<REDACTED>; cluster_name=company; >> column_index_cache_size_in_kb=2; column_index_size_in_kb=64; >> commit_failure_policy=ignore; commitlog_compression=null; >> commitlog_directory=/mnt/cassandra/commitlog; >> commitlog_max_compression_buffers_in_pool=3; >> commitlog_periodic_queue_size=-1; >> commitlog_segment_size_in_mb=32; commitlog_sync=periodic; >> commitlog_sync_batch_window_in_ms=NaN; >> commitlog_sync_period_in_ms=10000; >> commitlog_total_space_in_mb=null; >> compaction_large_partition_warning_threshold_mb=100; >> compaction_throughput_mb_per_sec=16; >> concurrent_compactors=null; concurrent_counter_writes=32; >> concurrent_materialized_view_writes=32; concurrent_reads=32; >> concurrent_replicates=null; concurrent_writes=32; >> counter_cache_keys_to_save=2147483647; >> counter_cache_save_period=7200; >> counter_cache_size_in_mb=null; >> counter_write_request_timeout_in_ms=6000000; >> credentials_cache_max_entries=1000; >> credentials_update_interval_in_ms=-1; >> credentials_validity_in_ms=2000; cross_node_timeout=false; >> data_file_directories=[Ljava.lang.String;@223f3642; >> disk_access_mode=auto; disk_failure_policy=ignore; >> disk_optimization_estimate_percentile=0.95; >> disk_optimization_page_cross_chance=0.1; >> disk_optimization_strategy=ssd; dynamic_snitch=true; >> dynamic_snitch_badness_threshold=0.1; >> dynamic_snitch_reset_interval_in_ms=600000; >> dynamic_snitch_update_interval_in_ms=100; >> enable_scripted_user_defined_functions=false; >> enable_user_defined_functions=false; >> enable_user_defined_functions_threads=true; >> encryption_options=null; endpoint_snitch=SimpleSnitch; >> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; >> gc_warn_threshold_in_ms=1000; >> hinted_handoff_disabled_datacenters=[]; >> hinted_handoff_enabled=true; >> hinted_handoff_throttle_in_kb=1024; hints_compression=null; >> hints_directory=/mnt/cassandra/hints; >> hints_flush_period_in_ms=10000; incremental_backups=false; >> index_interval=null; index_summary_capacity_in_mb=null; >> index_summary_resize_interval_in_minutes=60; >> initial_token=null; >> inter_dc_stream_throughput_outbound_megabits_per_sec=200; >> inter_dc_tcp_nodelay=false; internode_authenticator=null; >> internode_compression=dc; >> internode_recv_buff_size_in_bytes=0; >> internode_send_buff_size_in_bytes=0; >> key_cache_keys_to_save=2147483647; >> key_cache_save_period=14400; key_cache_size_in_mb=null; >> listen_address=10.100.100.213; listen_interface=null; >> listen_interface_prefer_ipv6=false; >> listen_on_broadcast_address=false; >> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; >> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; >> max_streaming_retries=3; max_value_size_in_mb=256; >> memtable_allocation_type=heap_buffers; >> memtable_cleanup_threshold=null; memtable_flush_writers=0; >> memtable_heap_space_in_mb=null; >> memtable_offheap_space_in_mb=null; >> min_free_space_per_drive_in_mb=50; >> native_transport_max_concurrent_connections=-1; >> native_transport_max_concurrent_connections_per_ip=-1; >> native_transport_max_frame_size_in_mb=256; >> native_transport_max_threads=128; native_transport_port=9042; >> native_transport_port_ssl=null; num_tokens=256; >> otc_coalescing_strategy=TIMEHORIZON; >> otc_coalescing_window_us=200; >> partitioner=org.apache.cassandra.dht.Murmur3Partitioner; >> permissions_cache_max_entries=1000; >> permissions_update_interval_in_ms=-1; >> permissions_validity_in_ms=2000; phi_convict_threshold=8.0; >> prepared_statements_cache_size_mb=null; >> range_request_timeout_in_ms=6000000; >> read_request_timeout_in_ms=6000000; >> request_scheduler=org.apache.cassandra.scheduler.NoScheduler; >> request_scheduler_id=null; request_scheduler_options=null; >> request_timeout_in_ms=6000000; >> role_manager=CassandraRoleManager; >> roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; >> roles_validity_in_ms=2000; >> row_cache_class_name=org.apache.cassandra.cache.OHCProvider; >> row_cache_keys_to_save=2147483647; row_cache_save_period=0; >> row_cache_size_in_mb=0; rpc_address=10.100.100.213; >> rpc_interface=null; rpc_interface_prefer_ipv6=false; >> rpc_keepalive=true; rpc_listen_backlog=50; >> rpc_max_threads=2147483647; rpc_min_threads=16; >> rpc_port=9160; rpc_recv_buff_size_in_bytes=null; >> rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; >> saved_caches_directory=/mnt/cassandra/saved_caches; >> >> seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.100.100.19, >> 10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52, >> 10.100.1000.213}; server_encryption_options=<REDACTED>; >> slow_query_log_timeout_in_ms=6000000; >> snapshot_before_compaction=false; ssl_storage_port=7001; >> sstable_preemptive_open_interval_in_mb=50; >> start_native_transport=true; start_rpc=false; >> storage_port=7000; >> stream_throughput_outbound_megabits_per_sec=200; >> streaming_keep_alive_period_in_secs=300; >> streaming_socket_timeout_in_ms=86400000; >> thrift_framed_transport_size_in_mb=15; >> thrift_max_message_length_in_mb=16; >> thrift_prepared_statements_cache_size_mb=null; >> tombstone_failure_threshold=100000; >> tombstone_warn_threshold=1000; tracetype_query_ttl=86400; >> tracetype_repair_ttl=604800; >> >> transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@38c5cc4c; >> trickle_fsync=false; trickle_fsync_interval_in_kb=10240; >> truncate_request_timeout_in_ms=6000000; >> unlogged_batch_across_partitions_warn_threshold=10; >> user_defined_function_fail_timeout=1500; >> user_defined_function_warn_timeout=500; >> user_function_timeout_policy=die; windows_timer_interval=1; >> write_request_timeout_in_ms=6000000] >> >> Thanks >> >> >> On 04/06/2017 11:30 AM, benjamin roth wrote: >>> Have you checked the effective limits of a running CS process? >>> Is CS run as Cassandra? Just to rule out missing file perms. >>> >>> >>> Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha" >>> <cogumelosmaravi...@sapo.pt >>> <mailto:cogumelosmaravi...@sapo.pt>>: >>> >>> From cassandra.yaml: >>> >>> hints_directory: /mnt/cassandra/hints >>> data_file_directories: >>> - /mnt/cassandra/data >>> commitlog_directory: /mnt/cassandra/commitlog >>> saved_caches_directory: /mnt/cassandra/saved_caches >>> >>> drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 mnt/ >>> >>> drwxr-xr-x 6 cassandra cassandra 68 Apr 5 16:17 ./ >>> drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 ../ >>> drwxr-xr-x 2 cassandra cassandra 80 Apr 6 10:07 commitlog/ >>> drwxr-xr-x 8 cassandra cassandra 124 Apr 5 16:17 data/ >>> drwxr-xr-x 2 cassandra cassandra 72 Apr 5 16:20 hints/ >>> drwxr-xr-x 2 cassandra cassandra 49 Apr 5 20:17 >>> saved_caches/ >>> >>> cassand+ 2267 1 99 10:18 ? 00:02:56 java >>> -Xloggc:/var/log/cassandra/gc.log -ea >>> -XX:+UseThreadPriorities -XX:Threa... >>> >>> /dev/mapper/um_vg-xfs_lv 885G 27G 858G 4% /mnt >>> >>> On /etc/security/limits.conf >>> >>> * - memlock unlimited >>> * - nofile 100000 >>> * - nproc 32768 >>> * - as unlimited >>> >>> On /etc/security/limits.d/cassandra.conf >>> >>> cassandra - memlock unlimited >>> cassandra - nofile 100000 >>> cassandra - as unlimited >>> cassandra - nproc 32768 >>> >>> On /etc/sysctl.conf >>> >>> vm.max_map_count = 1048575 >>> >>> On /etc/systcl.d/cassanda.conf >>> >>> vm.max_map_count = 1048575 >>> net.ipv4.tcp_keepalive_time=600 >>> >>> On /etc/pam.d/su >>> ... >>> session required pam_limits.so >>> ... >>> >>> Distro is the currently Ubuntu LTS. >>> Thanks >>> >>> >>> >>> On 04/06/2017 10:39 AM, benjamin roth wrote: >>>> Cassandra cannot write an SSTable to disk. Are you sure >>>> the disk/volume where SSTables reside (normally >>>> /var/lib/cassandra/data) is writeable for the CS user >>>> and has enough free space? >>>> The CDC warning also implies that. >>>> The other warnings indicate you are probably not >>>> running CS as root and you did not set an appropriate >>>> limit for max open files. Running out of open files can >>>> also be a reason for the IO error. >>>> >>>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha >>>> <cogumelosmaravi...@sapo.pt >>>> <mailto:cogumelosmaravi...@sapo.pt>>: >>>> >>>> Hi list, >>>> >>>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All >>>> instances type >>>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS >>>> formated 885G. I have >>>> one node that is always dieing and I don't >>>> understand why. Can anyone >>>> give me some hints please. All nodes using the same >>>> configuration. >>>> >>>> Thanks in advance. >>>> >>>> INFO [IndexSummaryManager:1] 2017-04-06 05:22:18,352 >>>> IndexSummaryRedistribution.java:75 - Redistributing >>>> index summaries >>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MemtablePostFlush:22,5,main] >>>> org.apache.cassandra.io >>>> <http://org.apache.cassandra.io>.FSWriteError: >>>> java.io.IOException: Input/output >>>> error >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> ~[na:1.8.0_121] >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> [na:1.8.0_121] >>>> at >>>> >>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) >>>> [apache-cassandra-3.10.jar:3.10] >>>> at java.lang.Thread.run(Thread.java:745) >>>> ~[na:1.8.0_121] >>>> Caused by: java.io.IOException: Input/output error >>>> at sun.nio.ch.FileDispatcherImpl.force0(Native >>>> Method) ~[na:1.8.0_121] >>>> at >>>> >>>> sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) >>>> ~[na:1.8.0_121] >>>> at >>>> sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) >>>> ~[na:1.8.0_121] >>>> at >>>> >>>> org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> ... 15 common frames omitted >>>> INFO [IndexSummaryManager:1] 2017-04-06 06:22:18,366 >>>> IndexSummaryRedistribution.java:75 - Redistributing >>>> index summaries >>>> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MemtablePostFlush:31,5,main] >>>> org.apache.cassandra.io >>>> <http://org.apache.cassandra.io>.FSWriteError: >>>> java.io.IOException: Input/output >>>> error >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> org.apache.cassandra.io >>>> >>>> <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> ~[na:1.8.0_121] >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> [na:1.8.0_121] >>>> at >>>> >>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) >>>> [apache-cassandra-3.10.jar:3.10] >>>> at java.lang.Thread.run(Thread.java:745) >>>> ~[na:1.8.0_121] >>>> Caused by: java.io.IOException: Input/output error >>>> at sun.nio.ch.FileDispatcherImpl.force0(Native >>>> Method) ~[na:1.8.0_121] >>>> at >>>> >>>> sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) >>>> ~[na:1.8.0_121] >>>> at >>>> sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) >>>> ~[na:1.8.0_121] >>>> at >>>> >>>> org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> at >>>> >>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169) >>>> ~[apache-cassandra-3.10.jar:3.10] >>>> ... 15 common frames omitted >>>> INFO [main] 2017-04-06 07:11:57,289 >>>> YamlConfigurationLoader.java:89 - >>>> Configuration location: >>>> file:/etc/cassandra/cassandra.yaml >>>> >>>> >>>> Some ERRORs messages: >>>> >>>> ERROR [MemtablePostFlush:2] 2017-04-05 23:35:46,339 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MemtablePostFlush:2,5,main] >>>> ERROR [MemtablePostFlush:3] 2017-04-05 23:44:08,471 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MemtablePostFlush:3,5,main] >>>> ERROR [MemtablePostFlush:4] 2017-04-05 23:54:41,224 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MemtablePostFlush:4,5,main] >>>> ERROR [MessagingService-Incoming-/10.0.120.52 >>>> <http://10.0.120.52>] 2017-04-06 03:19:13,453 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MessagingService-Incoming-/10.0.120.52 >>>> <http://10.0.120.52>,5,main] >>>> ERROR [epollEventLoopGroup-2-6] 2017-04-06 03:24:41,006 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[epollEventLoopGroup-2-6,10,main] >>>> ERROR [Native-Transport-Requests-36] 2017-04-06 >>>> 03:25:45,915 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-49] 2017-04-06 >>>> 03:25:45,915 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [IndexSummaryManager:1] 2017-04-06 03:25:45,915 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-69] 2017-04-06 >>>> 03:25:45,916 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-46] 2017-04-06 >>>> 03:26:18,465 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [SharedPool-Worker-136] 2017-04-06 03:26:18,465 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-156] 2017-04-06 >>>> 03:26:18,465 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [SharedPool-Worker-92] 2017-04-06 03:26:24,696 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-48] 2017-04-06 >>>> 03:26:24,696 ?:? - JVM >>>> state determined to be unstable. Exiting >>>> forcefully due to: >>>> ERROR [Native-Transport-Requests-66] 2017-04-06 >>>> 03:26:55,808 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-77] 2017-04-06 >>>> 03:26:55,808 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [GossipTasks:1] 2017-04-06 03:26:55,808 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-133] 2017-04-06 >>>> 03:26:55,808 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-135] 2017-04-06 >>>> 03:26:55,808 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [ScheduledFastTasks:1] 2017-04-06 03:26:55,808 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-70] 2017-04-06 >>>> 03:27:11,569 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [IndexSummaryManager:1] 2017-04-06 03:27:17,821 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[IndexSummaryManager:1,1,main] >>>> ERROR [Native-Transport-Requests-103] 2017-04-06 >>>> 03:27:24,049 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-69] 2017-04-06 >>>> 03:27:24,049 >>>> SEPWorker.java:145 - Failed to execute task, >>>> unexpected exception killed >>>> worker: {} >>>> ERROR [SharedPool-Worker-98] 2017-04-06 03:27:24,049 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [MessagingService-Incoming-/10.0.120.52 >>>> <http://10.0.120.52>] 2017-04-06 03:27:55,079 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [epollEventLoopGroup-2-5] 2017-04-06 03:27:55,079 >>>> JVMStabilityInspector.java:142 - JVM state >>>> determined to be unstable. >>>> Exiting forcefully due to: >>>> ERROR [Native-Transport-Requests-64] 2017-04-06 >>>> 03:28:43,285 >>>> SEPWorker.java:145 - Failed to execute task, >>>> unexpected exception killed >>>> worker: {} >>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MemtablePostFlush:22,5,main] >>>> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525 >>>> CassandraDaemon.java:229 - Exception in thread >>>> Thread[MemtablePostFlush:31,5,main] >>>> >>>> Also some WARNs: >>>> >>>> WARN [main] 2017-04-06 09:26:49,725 >>>> CLibrary.java:178 - Unable to lock >>>> JVM memory (ENOMEM). This can result in part of the >>>> JVM being swapped >>>> out, especially with mmapped I/O enabled. Increase >>>> RLIMIT_MEMLOCK or run >>>> Cassandra as root. >>>> >>>> WARN [main] 2017-04-06 09:25:07,355 >>>> StartupChecks.java:157 - JMX is not >>>> enabled to receive remote connections. Please see >>>> cassandra-env.sh for >>>> more info. >>>> >>>> WARN [main] 2017-04-06 09:25:07,369 >>>> SigarLibrary.java:174 - Cassandra >>>> server running in degraded mode. Is swap disabled? >>>> : true, Address >>>> space adequate? : true, nofile limit adequate? : >>>> false, nproc limit >>>> adequate? : true >>>> >>>> WARN [main] 2017-04-06 09:25:07,091 >>>> DatabaseDescriptor.java:493 - Small >>>> cdc volume detected at /var/lib/cassandra/cdc_raw; >>>> setting >>>> cdc_total_space_in_mb to 2502. You can override >>>> this in cassandra.yaml >>>> >>>> >>>> >>> >>> >> >> > > > -- > > >
