"system_auth" not my table.
On 04/11/2017 07:12 AM, Oskar Kjellin wrote: > You changed to 6 nodes because you were running out of disk? But you > still replicate 100% to all so you don't gain anything > > > > On 10 Apr 2017, at 13:48, Cogumelos Maravilha > <cogumelosmaravi...@sapo.pt <mailto:cogumelosmaravi...@sapo.pt>> wrote: > >> No. >> >> nodetool status, nodetool describecluster also nodetool ring shows a >> correct cluster. >> >> Not all nodes needs to be a seed, but can be. >> >> I had also ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : >> 'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false; >> >> And the first command on the new node was |nodetool repair system_auth| >> >> >> On 04/10/2017 12:37 PM, Chris Mawata wrote: >>> Notice >>> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85, >>> 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213}; >>> >>> Why do you have all six of your nodes as seeds? is it possible that >>> the last one you added used itself as the seed and is isolated? >>> >>> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha >>> <cogumelosmaravi...@sapo.pt <mailto:cogumelosmaravi...@sapo.pt>> wrote: >>> >>> Yes C* is running as cassandra: >>> >>> cassand+ 2267 1 99 10:18 ? 00:02:56 java >>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities >>> -XX:Threa... >>> >>> INFO [main] 2017-04-06 10:35:42,956 Config.java:474 - Node >>> configuration:[allocate_tokens_for_keyspace=null; >>> authenticator=PasswordAuthenticator; >>> authorizer=CassandraAuthorizer; auto_bootstrap=true; >>> auto_snapshot=true; back_pressure_enabled=false; >>> back_pressure_strategy=org.apache.cassandra.net >>> <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9, >>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; >>> batch_size_warn_threshold_in_kb=5; >>> batchlog_replay_throttle_in_kb=1024; broadcast_address=null; >>> broadcast_rpc_address=null; >>> buffer_pool_use_heap_if_exhausted=true; >>> cas_contention_timeout_in_ms=6000000; cdc_enabled=false; >>> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; >>> cdc_total_space_in_mb=0; client_encryption_options=<REDACTED>; >>> cluster_name=company; column_index_cache_size_in_kb=2; >>> column_index_size_in_kb=64; commit_failure_policy=ignore; >>> commitlog_compression=null; >>> commitlog_directory=/mnt/cassandra/commitlog; >>> commitlog_max_compression_buffers_in_pool=3; >>> commitlog_periodic_queue_size=-1; >>> commitlog_segment_size_in_mb=32; commitlog_sync=periodic; >>> commitlog_sync_batch_window_in_ms=NaN; >>> commitlog_sync_period_in_ms=10000; >>> commitlog_total_space_in_mb=null; >>> compaction_large_partition_warning_threshold_mb=100; >>> compaction_throughput_mb_per_sec=16; concurrent_compactors=null; >>> concurrent_counter_writes=32; >>> concurrent_materialized_view_writes=32; concurrent_reads=32; >>> concurrent_replicates=null; concurrent_writes=32; >>> counter_cache_keys_to_save=2147483647; >>> counter_cache_save_period=7200; counter_cache_size_in_mb=null; >>> counter_write_request_timeout_in_ms=6000000; >>> credentials_cache_max_entries=1000; >>> credentials_update_interval_in_ms=-1; >>> credentials_validity_in_ms=2000; cross_node_timeout=false; >>> data_file_directories=[Ljava.lang.String;@223f3642; >>> disk_access_mode=auto; disk_failure_policy=ignore; >>> disk_optimization_estimate_percentile=0.95; >>> disk_optimization_page_cross_chance=0.1; >>> disk_optimization_strategy=ssd; dynamic_snitch=true; >>> dynamic_snitch_badness_threshold=0.1; >>> dynamic_snitch_reset_interval_in_ms=600000; >>> dynamic_snitch_update_interval_in_ms=100; >>> enable_scripted_user_defined_functions=false; >>> enable_user_defined_functions=false; >>> enable_user_defined_functions_threads=true; >>> encryption_options=null; endpoint_snitch=SimpleSnitch; >>> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; >>> gc_warn_threshold_in_ms=1000; >>> hinted_handoff_disabled_datacenters=[]; >>> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; >>> hints_compression=null; hints_directory=/mnt/cassandra/hints; >>> hints_flush_period_in_ms=10000; incremental_backups=false; >>> index_interval=null; index_summary_capacity_in_mb=null; >>> index_summary_resize_interval_in_minutes=60; initial_token=null; >>> inter_dc_stream_throughput_outbound_megabits_per_sec=200; >>> inter_dc_tcp_nodelay=false; internode_authenticator=null; >>> internode_compression=dc; internode_recv_buff_size_in_bytes=0; >>> internode_send_buff_size_in_bytes=0; >>> key_cache_keys_to_save=2147483647; key_cache_save_period=14400; >>> key_cache_size_in_mb=null; listen_address=10.100.100.213; >>> listen_interface=null; listen_interface_prefer_ipv6=false; >>> listen_on_broadcast_address=false; >>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; >>> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; >>> max_streaming_retries=3; max_value_size_in_mb=256; >>> memtable_allocation_type=heap_buffers; >>> memtable_cleanup_threshold=null; memtable_flush_writers=0; >>> memtable_heap_space_in_mb=null; >>> memtable_offheap_space_in_mb=null; >>> min_free_space_per_drive_in_mb=50; >>> native_transport_max_concurrent_connections=-1; >>> native_transport_max_concurrent_connections_per_ip=-1; >>> native_transport_max_frame_size_in_mb=256; >>> native_transport_max_threads=128; native_transport_port=9042; >>> native_transport_port_ssl=null; num_tokens=256; >>> otc_coalescing_strategy=TIMEHORIZON; >>> otc_coalescing_window_us=200; >>> partitioner=org.apache.cassandra.dht.Murmur3Partitioner; >>> permissions_cache_max_entries=1000; >>> permissions_update_interval_in_ms=-1; >>> permissions_validity_in_ms=2000; phi_convict_threshold=8.0; >>> prepared_statements_cache_size_mb=null; >>> range_request_timeout_in_ms=6000000; >>> read_request_timeout_in_ms=6000000; >>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler; >>> request_scheduler_id=null; request_scheduler_options=null; >>> request_timeout_in_ms=6000000; >>> role_manager=CassandraRoleManager; roles_cache_max_entries=1000; >>> roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; >>> row_cache_class_name=org.apache.cassandra.cache.OHCProvider; >>> row_cache_keys_to_save=2147483647; row_cache_save_period=0; >>> row_cache_size_in_mb=0; rpc_address=10.100.100.213; >>> rpc_interface=null; rpc_interface_prefer_ipv6=false; >>> rpc_keepalive=true; rpc_listen_backlog=50; >>> rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; >>> rpc_recv_buff_size_in_bytes=null; >>> rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; >>> saved_caches_directory=/mnt/cassandra/saved_caches; >>> >>> seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.100.100.19, >>> 10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52, >>> 10.100.1000.213}; server_encryption_options=<REDACTED>; >>> slow_query_log_timeout_in_ms=6000000; >>> snapshot_before_compaction=false; ssl_storage_port=7001; >>> sstable_preemptive_open_interval_in_mb=50; >>> start_native_transport=true; start_rpc=false; storage_port=7000; >>> stream_throughput_outbound_megabits_per_sec=200; >>> streaming_keep_alive_period_in_secs=300; >>> streaming_socket_timeout_in_ms=86400000; >>> thrift_framed_transport_size_in_mb=15; >>> thrift_max_message_length_in_mb=16; >>> thrift_prepared_statements_cache_size_mb=null; >>> tombstone_failure_threshold=100000; >>> tombstone_warn_threshold=1000; tracetype_query_ttl=86400; >>> tracetype_repair_ttl=604800; >>> >>> transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@38c5cc4c; >>> trickle_fsync=false; trickle_fsync_interval_in_kb=10240; >>> truncate_request_timeout_in_ms=6000000; >>> unlogged_batch_across_partitions_warn_threshold=10; >>> user_defined_function_fail_timeout=1500; >>> user_defined_function_warn_timeout=500; >>> user_function_timeout_policy=die; windows_timer_interval=1; >>> write_request_timeout_in_ms=6000000] >>> >>> Thanks >>> >>> >>> On 04/06/2017 11:30 AM, benjamin roth wrote: >>>> Have you checked the effective limits of a running CS process? >>>> Is CS run as Cassandra? Just to rule out missing file perms. >>>> >>>> >>>> Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha" >>>> <cogumelosmaravi...@sapo.pt <mailto:cogumelosmaravi...@sapo.pt>>: >>>> >>>> From cassandra.yaml: >>>> >>>> hints_directory: /mnt/cassandra/hints >>>> data_file_directories: >>>> - /mnt/cassandra/data >>>> commitlog_directory: /mnt/cassandra/commitlog >>>> saved_caches_directory: /mnt/cassandra/saved_caches >>>> >>>> drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 mnt/ >>>> >>>> drwxr-xr-x 6 cassandra cassandra 68 Apr 5 16:17 ./ >>>> drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 ../ >>>> drwxr-xr-x 2 cassandra cassandra 80 Apr 6 10:07 commitlog/ >>>> drwxr-xr-x 8 cassandra cassandra 124 Apr 5 16:17 data/ >>>> drwxr-xr-x 2 cassandra cassandra 72 Apr 5 16:20 hints/ >>>> drwxr-xr-x 2 cassandra cassandra 49 Apr 5 20:17 saved_caches/ >>>> >>>> cassand+ 2267 1 99 10:18 ? 00:02:56 java >>>> -Xloggc:/var/log/cassandra/gc.log -ea >>>> -XX:+UseThreadPriorities -XX:Threa... >>>> >>>> /dev/mapper/um_vg-xfs_lv 885G 27G 858G 4% /mnt >>>> >>>> On /etc/security/limits.conf >>>> >>>> * - memlock unlimited >>>> * - nofile 100000 >>>> * - nproc 32768 >>>> * - as unlimited >>>> >>>> On /etc/security/limits.d/cassandra.conf >>>> >>>> cassandra - memlock unlimited >>>> cassandra - nofile 100000 >>>> cassandra - as unlimited >>>> cassandra - nproc 32768 >>>> >>>> On /etc/sysctl.conf >>>> >>>> vm.max_map_count = 1048575 >>>> >>>> On /etc/systcl.d/cassanda.conf >>>> >>>> vm.max_map_count = 1048575 >>>> net.ipv4.tcp_keepalive_time=600 >>>> >>>> On /etc/pam.d/su >>>> ... >>>> session required pam_limits.so >>>> ... >>>> >>>> Distro is the currently Ubuntu LTS. >>>> Thanks >>>> >>>> >>>> >>>> On 04/06/2017 10:39 AM, benjamin roth wrote: >>>>> Cassandra cannot write an SSTable to disk. Are you sure >>>>> the disk/volume where SSTables reside (normally >>>>> /var/lib/cassandra/data) is writeable for the CS user and >>>>> has enough free space? >>>>> The CDC warning also implies that. >>>>> The other warnings indicate you are probably not running >>>>> CS as root and you did not set an appropriate limit for >>>>> max open files. Running out of open files can also be a >>>>> reason for the IO error. >>>>> >>>>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha >>>>> <cogumelosmaravi...@sapo.pt >>>>> <mailto:cogumelosmaravi...@sapo.pt>>: >>>>> >>>>> Hi list, >>>>> >>>>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All >>>>> instances type >>>>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS >>>>> formated 885G. I have >>>>> one node that is always dieing and I don't understand >>>>> why. Can anyone >>>>> give me some hints please. All nodes using the same >>>>> configuration. >>>>> >>>>> Thanks in advance. >>>>> >>>>> INFO [IndexSummaryManager:1] 2017-04-06 05:22:18,352 >>>>> IndexSummaryRedistribution.java:75 - Redistributing >>>>> index summaries >>>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MemtablePostFlush:22,5,main] >>>>> org.apache.cassandra.io >>>>> <http://org.apache.cassandra.io>.FSWriteError: >>>>> java.io.IOException: Input/output >>>>> error >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> ~[na:1.8.0_121] >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> [na:1.8.0_121] >>>>> at >>>>> >>>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) >>>>> [apache-cassandra-3.10.jar:3.10] >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> ~[na:1.8.0_121] >>>>> Caused by: java.io.IOException: Input/output error >>>>> at sun.nio.ch.FileDispatcherImpl.force0(Native >>>>> Method) ~[na:1.8.0_121] >>>>> at >>>>> >>>>> sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) >>>>> ~[na:1.8.0_121] >>>>> at >>>>> sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) >>>>> ~[na:1.8.0_121] >>>>> at >>>>> org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> ... 15 common frames omitted >>>>> INFO [IndexSummaryManager:1] 2017-04-06 06:22:18,366 >>>>> IndexSummaryRedistribution.java:75 - Redistributing >>>>> index summaries >>>>> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MemtablePostFlush:31,5,main] >>>>> org.apache.cassandra.io >>>>> <http://org.apache.cassandra.io>.FSWriteError: >>>>> java.io.IOException: Input/output >>>>> error >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> org.apache.cassandra.io >>>>> >>>>> <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> ~[na:1.8.0_121] >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> [na:1.8.0_121] >>>>> at >>>>> >>>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) >>>>> [apache-cassandra-3.10.jar:3.10] >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> ~[na:1.8.0_121] >>>>> Caused by: java.io.IOException: Input/output error >>>>> at sun.nio.ch.FileDispatcherImpl.force0(Native >>>>> Method) ~[na:1.8.0_121] >>>>> at >>>>> >>>>> sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) >>>>> ~[na:1.8.0_121] >>>>> at >>>>> sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) >>>>> ~[na:1.8.0_121] >>>>> at >>>>> org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> at >>>>> >>>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169) >>>>> ~[apache-cassandra-3.10.jar:3.10] >>>>> ... 15 common frames omitted >>>>> INFO [main] 2017-04-06 07:11:57,289 >>>>> YamlConfigurationLoader.java:89 - >>>>> Configuration location: file:/etc/cassandra/cassandra.yaml >>>>> >>>>> >>>>> Some ERRORs messages: >>>>> >>>>> ERROR [MemtablePostFlush:2] 2017-04-05 23:35:46,339 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MemtablePostFlush:2,5,main] >>>>> ERROR [MemtablePostFlush:3] 2017-04-05 23:44:08,471 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MemtablePostFlush:3,5,main] >>>>> ERROR [MemtablePostFlush:4] 2017-04-05 23:54:41,224 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MemtablePostFlush:4,5,main] >>>>> ERROR [MessagingService-Incoming-/10.0.120.52 >>>>> <http://10.0.120.52>] 2017-04-06 03:19:13,453 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MessagingService-Incoming-/10.0.120.52 >>>>> <http://10.0.120.52>,5,main] >>>>> ERROR [epollEventLoopGroup-2-6] 2017-04-06 03:24:41,006 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[epollEventLoopGroup-2-6,10,main] >>>>> ERROR [Native-Transport-Requests-36] 2017-04-06 >>>>> 03:25:45,915 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-49] 2017-04-06 >>>>> 03:25:45,915 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [IndexSummaryManager:1] 2017-04-06 03:25:45,915 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-69] 2017-04-06 >>>>> 03:25:45,916 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-46] 2017-04-06 >>>>> 03:26:18,465 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [SharedPool-Worker-136] 2017-04-06 03:26:18,465 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-156] 2017-04-06 >>>>> 03:26:18,465 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [SharedPool-Worker-92] 2017-04-06 03:26:24,696 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-48] 2017-04-06 >>>>> 03:26:24,696 ?:? - JVM >>>>> state determined to be unstable. Exiting forcefully >>>>> due to: >>>>> ERROR [Native-Transport-Requests-66] 2017-04-06 >>>>> 03:26:55,808 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-77] 2017-04-06 >>>>> 03:26:55,808 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [GossipTasks:1] 2017-04-06 03:26:55,808 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-133] 2017-04-06 >>>>> 03:26:55,808 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-135] 2017-04-06 >>>>> 03:26:55,808 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [ScheduledFastTasks:1] 2017-04-06 03:26:55,808 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-70] 2017-04-06 >>>>> 03:27:11,569 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [IndexSummaryManager:1] 2017-04-06 03:27:17,821 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[IndexSummaryManager:1,1,main] >>>>> ERROR [Native-Transport-Requests-103] 2017-04-06 >>>>> 03:27:24,049 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-69] 2017-04-06 >>>>> 03:27:24,049 >>>>> SEPWorker.java:145 - Failed to execute task, >>>>> unexpected exception killed >>>>> worker: {} >>>>> ERROR [SharedPool-Worker-98] 2017-04-06 03:27:24,049 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [MessagingService-Incoming-/10.0.120.52 >>>>> <http://10.0.120.52>] 2017-04-06 03:27:55,079 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [epollEventLoopGroup-2-5] 2017-04-06 03:27:55,079 >>>>> JVMStabilityInspector.java:142 - JVM state determined >>>>> to be unstable. >>>>> Exiting forcefully due to: >>>>> ERROR [Native-Transport-Requests-64] 2017-04-06 >>>>> 03:28:43,285 >>>>> SEPWorker.java:145 - Failed to execute task, >>>>> unexpected exception killed >>>>> worker: {} >>>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MemtablePostFlush:22,5,main] >>>>> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525 >>>>> CassandraDaemon.java:229 - Exception in thread >>>>> Thread[MemtablePostFlush:31,5,main] >>>>> >>>>> Also some WARNs: >>>>> >>>>> WARN [main] 2017-04-06 09:26:49,725 CLibrary.java:178 >>>>> - Unable to lock >>>>> JVM memory (ENOMEM). This can result in part of the >>>>> JVM being swapped >>>>> out, especially with mmapped I/O enabled. Increase >>>>> RLIMIT_MEMLOCK or run >>>>> Cassandra as root. >>>>> >>>>> WARN [main] 2017-04-06 09:25:07,355 >>>>> StartupChecks.java:157 - JMX is not >>>>> enabled to receive remote connections. Please see >>>>> cassandra-env.sh for >>>>> more info. >>>>> >>>>> WARN [main] 2017-04-06 09:25:07,369 >>>>> SigarLibrary.java:174 - Cassandra >>>>> server running in degraded mode. Is swap disabled? : >>>>> true, Address >>>>> space adequate? : true, nofile limit adequate? : >>>>> false, nproc limit >>>>> adequate? : true >>>>> >>>>> WARN [main] 2017-04-06 09:25:07,091 >>>>> DatabaseDescriptor.java:493 - Small >>>>> cdc volume detected at /var/lib/cassandra/cdc_raw; setting >>>>> cdc_total_space_in_mb to 2502. You can override this >>>>> in cassandra.yaml >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>
