Re: Node always dieing

[Cogumelos Maravilha]

"system_auth" not my table.


On 04/11/2017 07:12 AM, Oskar Kjellin wrote:
> You changed to 6 nodes because you were running out of disk? But you
> still replicate 100% to all so you don't gain anything 
>
>
>
> On 10 Apr 2017, at 13:48, Cogumelos Maravilha
> > wrote:
>
>> No.
>>
>> nodetool status, nodetool describecluster also nodetool ring shows a
>> correct cluster.
>>
>> Not all nodes needs to be a seed, but can be.
>>
>> I had also ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' :
>> 'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false;
>>
>> And the first command on the new node was  |nodetool repair system_auth|
>>
>>
>> On 04/10/2017 12:37 PM, Chris Mawata wrote:
>>> Notice
>>> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85,
>>> 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213};
>>>
>>> Why do you have all six of your nodes as seeds? is it possible that
>>> the last one you added used itself as the seed and is isolated?
>>>
>>> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha
>>> > wrote:
>>>
>>> Yes C* is running as cassandra:
>>>
>>> cassand+  2267 1 99 10:18 ?00:02:56 java
>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
>>> -XX:Threa...
>>>
>>> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>>> configuration:[allocate_tokens_for_keyspace=null;
>>> authenticator=PasswordAuthenticator;
>>> authorizer=CassandraAuthorizer; auto_bootstrap=true;
>>> auto_snapshot=true; back_pressure_enabled=false;
>>> back_pressure_strategy=org.apache.cassandra.net
>>> .RateBasedBackPressure{high_ratio=0.9,
>>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>>> batch_size_warn_threshold_in_kb=5;
>>> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
>>> broadcast_rpc_address=null;
>>> buffer_pool_use_heap_if_exhausted=true;
>>> cas_contention_timeout_in_ms=600; cdc_enabled=false;
>>> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
>>> cdc_total_space_in_mb=0; client_encryption_options=;
>>> cluster_name=company; column_index_cache_size_in_kb=2;
>>> column_index_size_in_kb=64; commit_failure_policy=ignore;
>>> commitlog_compression=null;
>>> commitlog_directory=/mnt/cassandra/commitlog;
>>> commitlog_max_compression_buffers_in_pool=3;
>>> commitlog_periodic_queue_size=-1;
>>> commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
>>> commitlog_sync_batch_window_in_ms=NaN;
>>> commitlog_sync_period_in_ms=1;
>>> commitlog_total_space_in_mb=null;
>>> compaction_large_partition_warning_threshold_mb=100;
>>> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
>>> concurrent_counter_writes=32;
>>> concurrent_materialized_view_writes=32; concurrent_reads=32;
>>> concurrent_replicates=null; concurrent_writes=32;
>>> counter_cache_keys_to_save=2147483647;
>>> counter_cache_save_period=7200; counter_cache_size_in_mb=null;
>>> counter_write_request_timeout_in_ms=600;
>>> credentials_cache_max_entries=1000;
>>> credentials_update_interval_in_ms=-1;
>>> credentials_validity_in_ms=2000; cross_node_timeout=false;
>>> data_file_directories=[Ljava.lang.String;@223f3642;
>>> disk_access_mode=auto; disk_failure_policy=ignore;
>>> disk_optimization_estimate_percentile=0.95;
>>> disk_optimization_page_cross_chance=0.1;
>>> disk_optimization_strategy=ssd; dynamic_snitch=true;
>>> dynamic_snitch_badness_threshold=0.1;
>>> dynamic_snitch_reset_interval_in_ms=60;
>>> dynamic_snitch_update_interval_in_ms=100;
>>> enable_scripted_user_defined_functions=false;
>>> enable_user_defined_functions=false;
>>> enable_user_defined_functions_threads=true;
>>> encryption_options=null; endpoint_snitch=SimpleSnitch;
>>> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
>>> gc_warn_threshold_in_ms=1000;
>>> hinted_handoff_disabled_datacenters=[];
>>> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
>>> hints_compression=null; hints_directory=/mnt/cassandra/hints;
>>> hints_flush_period_in_ms=1; incremental_backups=false;
>>> index_interval=null; index_summary_capacity_in_mb=null;
>>> index_summary_resize_interval_in_minutes=60; initial_token=null;
>>> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
>>> inter_dc_tcp_nodelay=false; internode_authenticator=null;
>>> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
>>> internode_send_buff_size_in_bytes=0;
>>> key_cache_keys_to_save=2147483647; key_cache_save_period=14400;
>>> key_cache_size_in_mb=null; listen_address=10.100.100.213;
>>> listen_interface=null; 

Re: Node always dieing

[Oskar Kjellin]

You changed to 6 nodes because you were running out of disk? But you still 
replicate 100% to all so you don't gain anything 



> On 10 Apr 2017, at 13:48, Cogumelos Maravilha  
> wrote:
> 
> No.
> 
> nodetool status, nodetool describecluster also nodetool ring shows a correct 
> cluster.
> 
> Not all nodes needs to be a seed, but can be.
> 
> I had also ALTER KEYSPACE system_auth WITH REPLICATION = {   'class' : 
> 'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false;
> 
> And the first command on the new node was  nodetool repair system_auth
> 
> 
>> On 04/10/2017 12:37 PM, Chris Mawata wrote:
>> Notice
>> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85, 10.100.100.185, 
>> 10.100.100.161, 10.100.100.52, 10.100.1000.213};
>> 
>> Why do you have all six of your nodes as seeds? is it possible that the last 
>> one you added used itself as the seed and is isolated?
>> 
>> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha 
>>  wrote:
>>> Yes C* is running as cassandra:
>>> 
>>> cassand+  2267 1 99 10:18 ?00:02:56 java 
>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
>>> 
>>> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node 
>>> configuration:[allocate_tokens_for_keyspace=null; 
>>> authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer; 
>>> auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; 
>>> back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
>>>  factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; 
>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; 
>>> broadcast_address=null; broadcast_rpc_address=null; 
>>> buffer_pool_use_heap_if_exhausted=true; 
>>> cas_contention_timeout_in_ms=600; cdc_enabled=false; 
>>> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; 
>>> cdc_total_space_in_mb=0; client_encryption_options=; 
>>> cluster_name=company; column_index_cache_size_in_kb=2; 
>>> column_index_size_in_kb=64; commit_failure_policy=ignore; 
>>> commitlog_compression=null; commitlog_directory=/mnt/cassandra/commitlog; 
>>> commitlog_max_compression_buffers_in_pool=3; 
>>> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; 
>>> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; 
>>> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=null;
>>>  compaction_large_partition_warning_threshold_mb=100; 
>>> compaction_throughput_mb_per_sec=16; concurrent_compactors=null; 
>>> concurrent_counter_writes=32; concurrent_materialized_view_writes=32; 
>>> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; 
>>> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; 
>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=600; 
>>> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; 
>>> credentials_validity_in_ms=2000; cross_node_timeout=false; 
>>> data_file_directories=[Ljava.lang.String;@223f3642; disk_access_mode=auto; 
>>> disk_failure_policy=ignore; disk_optimization_estimate_percentile=0.95; 
>>> disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; 
>>> dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; 
>>> dynamic_snitch_reset_interval_in_ms=60; 
>>> dynamic_snitch_update_interval_in_ms=100; 
>>> enable_scripted_user_defined_functions=false; 
>>> enable_user_defined_functions=false; 
>>> enable_user_defined_functions_threads=true; encryption_options=null; 
>>> endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; 
>>> gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; 
>>> hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; 
>>> hinted_handoff_throttle_in_kb=1024; hints_compression=null; 
>>> hints_directory=/mnt/cassandra/hints; hints_flush_period_in_ms=1; 
>>> incremental_backups=false; index_interval=null; 
>>> index_summary_capacity_in_mb=null; 
>>> index_summary_resize_interval_in_minutes=60; initial_token=null; 
>>> inter_dc_stream_throughput_outbound_megabits_per_sec=200; 
>>> inter_dc_tcp_nodelay=false; internode_authenticator=null; 
>>> internode_compression=dc; internode_recv_buff_size_in_bytes=0; 
>>> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; 
>>> key_cache_save_period=14400; key_cache_size_in_mb=null; 
>>> listen_address=10.100.100.213; listen_interface=null; 
>>> listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; 
>>> max_hint_window_in_ms=1080; max_hints_delivery_threads=2; 
>>> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; 
>>> max_streaming_retries=3; max_value_size_in_mb=256; 
>>> memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; 
>>> memtable_flush_writers=0; memtable_heap_space_in_mb=null; 
>>> 

RE: Node always dieing

[Hartmut Inerle]

Hi,

There is an invalid IP in the String.
.SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85, 10.100.100.185, 
10.100.100.161, 10.100.100.52, 10.100.1000.213};

Regards
  Hartmut


From: Chris Mawata [mailto:chris.maw...@gmail.com]
Sent: Montag, 10. April 2017 13:37
To: user@cassandra.apache.org
Subject: Re: Node always dieing

Notice
.SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85, 10.100.100.185, 
10.100.100.161, 10.100.100.52, 10.100.1000.213};
Why do you have all six of your nodes as seeds? is it possible that the last 
one you added used itself as the seed and is isolated?

On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha 
<cogumelosmaravi...@sapo.pt<mailto:cogumelosmaravi...@sapo.pt>> wrote:

Yes C* is running as cassandra:

cassand+  2267 1 99 10:18 ?00:02:56 java 
-Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node 
configuration:[allocate_tokens_for_keyspace=null; 
authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer; 
auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; 
back_pressure_strategy=org.apache.cassandra.net<http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9,
 factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; 
batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; 
broadcast_address=null; broadcast_rpc_address=null; 
buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=600; 
cdc_enabled=false; cdc_free_space_check_interval_ms=250; 
cdc_raw_directory=null; cdc_total_space_in_mb=0; 
client_encryption_options=; cluster_name=company; 
column_index_cache_size_in_kb=2; column_index_size_in_kb=64; 
commit_failure_policy=ignore; commitlog_compression=null; 
commitlog_directory=/mnt/cassandra/commitlog; 
commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; 
commitlog_segment_size_in_mb=32; commitlog_sync=periodic; 
commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=1; 
commitlog_total_space_in_mb=null; 
compaction_large_partition_warning_threshold_mb=100; 
compaction_throughput_mb_per_sec=16; concurrent_compactors=null; 
concurrent_counter_writes=32; concurrent_materialized_view_writes=32; 
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; 
counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; 
counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=600; 
credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; 
credentials_validity_in_ms=2000; cross_node_timeout=false; 
data_file_directories=[Ljava.lang.String;@223f3642; disk_access_mode=auto; 
disk_failure_policy=ignore; disk_optimization_estimate_percentile=0.95; 
disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; 
dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; 
dynamic_snitch_reset_interval_in_ms=60; 
dynamic_snitch_update_interval_in_ms=100; 
enable_scripted_user_defined_functions=false; 
enable_user_defined_functions=false; 
enable_user_defined_functions_threads=true; encryption_options=null; 
endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; 
gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; 
hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; 
hinted_handoff_throttle_in_kb=1024; hints_compression=null; 
hints_directory=/mnt/cassandra/hints; hints_flush_period_in_ms=1; 
incremental_backups=false; index_interval=null; 
index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; 
initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; 
inter_dc_tcp_nodelay=false; internode_authenticator=null; 
internode_compression=dc; internode_recv_buff_size_in_bytes=0; 
internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; 
key_cache_save_period=14400; key_cache_size_in_mb=null; 
listen_address=10.100.100.213; listen_interface=null; 
listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; 
max_hint_window_in_ms=1080; max_hints_delivery_threads=2; 
max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; 
max_streaming_retries=3; max_value_size_in_mb=256; 
memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; 
memtable_flush_writers=0; memtable_heap_space_in_mb=null; 
memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; 
native_transport_max_concurrent_connections=-1; 
native_transport_max_concurrent_connections_per_ip=-1; 
native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; 
native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; 
otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200; 
partitioner=org.apache.cassandra.dht.Murmur3Partitioner; 
permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; 
permissions_validity_in_ms=2000; phi_con

Re: Node always dieing

[Cogumelos Maravilha]

No.

nodetool status, nodetool describecluster also nodetool ring shows a
correct cluster.

Not all nodes needs to be a seed, but can be.

I had also ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' :
'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false;

And the first command on the new node was  |nodetool repair system_auth|


On 04/10/2017 12:37 PM, Chris Mawata wrote:
> Notice
> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85,
> 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213};
>
> Why do you have all six of your nodes as seeds? is it possible that
> the last one you added used itself as the seed and is isolated?
>
> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha
> > wrote:
>
> Yes C* is running as cassandra:
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
> -XX:Threa...
>
> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
> configuration:[allocate_tokens_for_keyspace=null;
> authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_bootstrap=true;
> auto_snapshot=true; back_pressure_enabled=false;
> back_pressure_strategy=org.apache.cassandra.net
> .RateBasedBackPressure{high_ratio=0.9,
> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5;
> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
> broadcast_rpc_address=null;
> buffer_pool_use_heap_if_exhausted=true;
> cas_contention_timeout_in_ms=600; cdc_enabled=false;
> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
> cdc_total_space_in_mb=0; client_encryption_options=;
> cluster_name=company; column_index_cache_size_in_kb=2;
> column_index_size_in_kb=64; commit_failure_policy=ignore;
> commitlog_compression=null;
> commitlog_directory=/mnt/cassandra/commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1;
> commitlog_total_space_in_mb=null;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
> concurrent_counter_writes=32;
> concurrent_materialized_view_writes=32; concurrent_reads=32;
> concurrent_replicates=null; concurrent_writes=32;
> counter_cache_keys_to_save=2147483647;
> counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> counter_write_request_timeout_in_ms=600;
> credentials_cache_max_entries=1000;
> credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@223f3642;
> disk_access_mode=auto; disk_failure_policy=ignore;
> disk_optimization_estimate_percentile=0.95;
> disk_optimization_page_cross_chance=0.1;
> disk_optimization_strategy=ssd; dynamic_snitch=true;
> dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100;
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false;
> enable_user_defined_functions_threads=true;
> encryption_options=null; endpoint_snitch=SimpleSnitch;
> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
> gc_warn_threshold_in_ms=1000;
> hinted_handoff_disabled_datacenters=[];
> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> hints_compression=null; hints_directory=/mnt/cassandra/hints;
> hints_flush_period_in_ms=1; incremental_backups=false;
> index_interval=null; index_summary_capacity_in_mb=null;
> index_summary_resize_interval_in_minutes=60; initial_token=null;
> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
> inter_dc_tcp_nodelay=false; internode_authenticator=null;
> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
> internode_send_buff_size_in_bytes=0;
> key_cache_keys_to_save=2147483647; key_cache_save_period=14400;
> key_cache_size_in_mb=null; listen_address=10.100.100.213;
> listen_interface=null; listen_interface_prefer_ipv6=false;
> listen_on_broadcast_address=false; max_hint_window_in_ms=1080;
> max_hints_delivery_threads=2; max_hints_file_size_in_mb=128;
> max_mutation_size_in_kb=null; max_streaming_retries=3;
> max_value_size_in_mb=256; memtable_allocation_type=heap_buffers;
> memtable_cleanup_threshold=null; memtable_flush_writers=0;
> memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null;
> min_free_space_per_drive_in_mb=50;
> native_transport_max_concurrent_connections=-1;
> 

Re: Node always dieing

[Chris Mawata]

Notice
.SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85, 10.100.100.185,
10.100.100.161, 10.100.100.52, 10.100.1000.213};

Why do you have all six of your nodes as seeds? is it possible that the
last one you added used itself as the seed and is isolated?

On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha <
cogumelosmaravi...@sapo.pt> wrote:

> Yes C* is running as cassandra:
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
> -XX:Threa...
>
> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
> configuration:[allocate_tokens_for_keyspace=null; 
> authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_bootstrap=true; auto_snapshot=true;
> back_pressure_enabled=false; back_pressure_strategy=org.
> apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5,
> flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> broadcast_address=null; broadcast_rpc_address=null; 
> buffer_pool_use_heap_if_exhausted=true;
> cas_contention_timeout_in_ms=600; cdc_enabled=false;
> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
> cdc_total_space_in_mb=0; client_encryption_options=;
> cluster_name=company; column_index_cache_size_in_kb=2;
> column_index_size_in_kb=64; commit_failure_policy=ignore;
> commitlog_compression=null; commitlog_directory=/mnt/cassandra/commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=null;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32;
> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=600;
> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@223f3642;
> disk_access_mode=auto; disk_failure_policy=ignore;
> disk_optimization_estimate_percentile=0.95; 
> disk_optimization_page_cross_chance=0.1;
> disk_optimization_strategy=ssd; dynamic_snitch=true;
> dynamic_snitch_badness_threshold=0.1; 
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100; 
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false; 
> enable_user_defined_functions_threads=true;
> encryption_options=null; endpoint_snitch=SimpleSnitch;
> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
> gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[];
> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> hints_compression=null; hints_directory=/mnt/cassandra/hints;
> hints_flush_period_in_ms=1; incremental_backups=false;
> index_interval=null; index_summary_capacity_in_mb=null;
> index_summary_resize_interval_in_minutes=60; initial_token=null;
> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
> inter_dc_tcp_nodelay=false; internode_authenticator=null;
> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
> key_cache_save_period=14400; key_cache_size_in_mb=null;
> listen_address=10.100.100.213; listen_interface=null;
> listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
> max_hint_window_in_ms=1080; max_hints_delivery_threads=2;
> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
> max_streaming_retries=3; max_value_size_in_mb=256;
> memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null;
> memtable_flush_writers=0; memtable_heap_space_in_mb=null;
> memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50;
> native_transport_max_concurrent_connections=-1; native_transport_max_
> concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256;
> native_transport_max_threads=128; native_transport_port=9042;
> native_transport_port_ssl=null; num_tokens=256; 
> otc_coalescing_strategy=TIMEHORIZON;
> otc_coalescing_window_us=200; 
> partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1;
> permissions_validity_in_ms=2000; phi_convict_threshold=8.0;
> prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=600;
> read_request_timeout_in_ms=600; request_scheduler=org.apache.
> cassandra.scheduler.NoScheduler; request_scheduler_id=null;
> request_scheduler_options=null; request_timeout_in_ms=600;
> role_manager=CassandraRoleManager; 

Re: Node always dieing

[Cogumelos Maravilha]

There's a tweak.

I've forgot to put this in the new instance:

At /lib/udev/rules.d/

|cat ||40-vm-hotadd.rules**||# On Hyper-V and Xen Virtual Machines we
want to add memory and cpus as soon as they appear|

|ATTR{[dmi/id]sys_vendor}=="Microsoft Corporation",
ATTR{[dmi/id]product_name}=="Virtual Machine", GOTO="vm_hotadd_apply"|
|ATTR{[dmi/id]sys_vendor}=="Xen", GOTO="vm_hotadd_apply"|
|GOTO="vm_hotadd_end"|
|LABEL="vm_hotadd_apply"|
|# Memory hotadd request|
|#SUBSYSTEM=="memory", ACTION=="add",
DEVPATH=="/devices/system/memory/memory[0-9]*", TEST=="state",
ATTR{state}="online"|
|# CPU hotadd request|
|SUBSYSTEM=="cpu", ACTION=="add",
DEVPATH=="/devices/system/cpu/cpu[0-9]*", TEST=="online", ATTR{online}="1"|
|LABEL="vm_hotadd_end"|
|#SUBSYSTEM=="memory", ACTION=="add",
DEVPATH=="/devices/system/memory/memory[0-9]*", TEST=="state",
ATTR{state}="online"|

Don't ask from where this code came from! Sharing workarounds.
On 04/06/2017 06:13 PM, Carlos Rolo wrote:
> i3 are having those issues more than the other instances it seems. Not
> the first report I heard about.
> Regards,
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> _linkedin.com/in/carlosjuzarterolo
> _
> Mobile: +351 918 918 100
> www.pythian.com 
> On Thu, Apr 6, 2017 at 5:36 PM, Cogumelos Maravilha
> > wrote:
>
> Yes but this time I going to give lots of time between killing and
> pickup.
>
> Thanks a lot.
> On 04/06/2017 05:31 PM, Avi Kivity wrote:
>>
>> Your disk is bad.  Kill that instance and hope someone else gets it.
>>
>> On 04/06/2017 07:27 PM, Cogumelos Maravilha wrote:
>>>
>>> Interesting
>>>
>>> [  720.693768] blk_update_request: I/O error, dev nvme0n1,
>>> sector 1397303056 [  750.698840] blk_update_request: I/O error,
>>> dev nvme0n1, sector 1397303080 [ 1416.202103]
>>> blk_update_request: I/O error, dev nvme0n1, sector 1397303080
>>>
>>> On 04/06/2017 05:26 PM, Avi Kivity wrote:

 Is there anything in dmesg?

 On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:
>
> Now dies and restart (systemd) without logging why
>
> system.log
>
> INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
> AuthCache.java:172 - (Re)initializing RolesCache (validity
> period /update interval/max entries) (2000/2000/1000) INFO 
> [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
> - Configuration location: file:/etc/cassandra/cassandra. yaml
>
> debug.log DEBUG [GossipStage:1] 2017-04-06 16:16:56,272
> FailureDetector.java:457 - Ignoring interval time of
> 2496703934 for /10.100.120.52  DEBUG
> [GossipStage:1] 2017-04-06 16:16:59,090
> FailureDetector.java:457 - Ignoring interval time of
> 2818071981 for /10.100.120.161  INFO 
> [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
> - Configuration location: file:/etc/cassandra/cassandra.yaml
> DEBUG [main] 2017-04-06 16:17:42,540
> YamlConfigurationLoader.java:108 - Loading settings from
> file:/etc/cassandra/cassandra.yaml
> On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
>> find //mnt/cassandra// \! -user cassandra
>> nothing
>>
>> I've found some "strange" solutions on Internet
>> chmod -R 2777 /tmp
>> chmod -R 2775 cassandra folder
>>
>> Lets give some time to see the result
>> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>>> All it takes is one frustrated `sudo cassandra` run. Checking only 
>>> the
>>> top level directory ownership is insufficient, since root could own
>>> files/dirs created below the top level. Find all files not owned by 
>>> user
>>> cassandra:  `find //mnt/cassandra// \! -user cassandra`
>>>
>>> Just another thought.
>>>
>>> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
 From cassandra.yaml:

 hints_directory: /mnt/cassandra/hints
 data_file_directories:
 - /mnt/cassandra/data
 commitlog_directory: /mnt/cassandra/commitlog
 saved_caches_directory: /mnt/cassandra/saved_caches

 drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/

 drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
 drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
 drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
 drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
 drwxr-xr-x 2 cassandra 

Re: Node always dieing

[Carlos Rolo]

i3 are having those issues more than the other instances it seems. Not the
first report I heard about.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Thu, Apr 6, 2017 at 5:36 PM, Cogumelos Maravilha <
cogumelosmaravi...@sapo.pt> wrote:

> Yes but this time I going to give lots of time between killing and pickup.
> Thanks a lot.
>
>
> On 04/06/2017 05:31 PM, Avi Kivity wrote:
>
> Your disk is bad.  Kill that instance and hope someone else gets it.
>
> On 04/06/2017 07:27 PM, Cogumelos Maravilha wrote:
>
> Interesting
>
> [  720.693768] blk_update_request: I/O error, dev nvme0n1, sector
> 1397303056
> [  750.698840] blk_update_request: I/O error, dev nvme0n1, sector
> 1397303080
> [ 1416.202103] blk_update_request: I/O error, dev nvme0n1, sector
> 1397303080
>
> On 04/06/2017 05:26 PM, Avi Kivity wrote:
>
> Is there anything in dmesg?
>
> On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:
>
> Now dies and restart (systemd) without logging why
>
> system.log
>
> INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
> AuthCache.java:172 - (Re)initializing RolesCache (validity period
> /update interval/max entries) (2000/2000/1000)
> INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 -
> Configuration location: file:/etc/cassandra/cassandra.
> yaml
>
> debug.log
> DEBUG [GossipStage:1] 2017-04-06 16:16:56,272 FailureDetector.java:457 -
> Ignoring interval time of 2496703934 for /10.100.120.52
> DEBUG [GossipStage:1] 2017-04-06 16:16:59,090 FailureDetector.java:457 -
> Ignoring interval time of 2818071981 for /10.100.120.161
> INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 -
> Configuration location: file:/etc/cassandra/cassandra.yaml
> DEBUG [main] 2017-04-06 16:17:42,540 YamlConfigurationLoader.java:108 -
> Loading settings from file:/etc/cassandra/cassandra.yaml
>
>
> On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
>
> find */mnt/cassandra/* \! -user cassandra
> nothing
>
> I've found some "strange" solutions on Internet
> chmod -R 2777 /tmp
> chmod -R 2775 cassandra folder
>
> Lets give some time to see the result
>
>
> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>
> All it takes is one frustrated `sudo cassandra` run. Checking only the
> top level directory ownership is insufficient, since root could own
> files/dirs created below the top level. Find all files not owned by user
> cassandra:  `find */mnt/cassandra/* \! -user cassandra`
>
> Just another thought.
>
> --
> Michael
>
>
> On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>
> From cassandra.yaml:
>
> hints_directory: /mnt/cassandra/hints
> data_file_directories:
> - /mnt/cassandra/data
> commitlog_directory: /mnt/cassandra/commitlog
> saved_caches_directory: /mnt/cassandra/saved_caches
>
> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>
> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
>
> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>
> On /etc/security/limits.conf
>
> *   -   memlock  unlimited
> *   -  nofile  10
> *   -  nproc  32768
> *   -  as   unlimited
>
> On /etc/security/limits.d/cassandra.conf
>
> cassandra  -  memlock  unlimited
> cassandra  -  nofile   10
> cassandra  -  as   unlimited
> cassandra  -  nproc32768
>
> On /etc/sysctl.conf
>
> vm.max_map_count = 1048575
>
> On /etc/systcl.d/cassanda.conf
>
> vm.max_map_count = 1048575
> net.ipv4.tcp_keepalive_time=600
>
> On /etc/pam.d/su
> ...
> sessionrequired   pam_limits.so
> ...
>
> Distro is the currently Ubuntu LTS.
> Thanks
>
>
> On 04/06/2017 10:39 AM, benjamin roth wrote:
>
> Cassandra cannot write an SSTable to disk. Are you sure the
> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
> is writeable for the CS user and has enough free space?
> The CDC warning also implies that.
> The other warnings indicate you are probably not running CS as root
> and you did not set an appropriate limit for max open files. Running
> out of open files can also be a reason for the IO error.
>
> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>  
> >:
>
> Hi list,
>
> I'm using C* 3.10 in a 6 nodes cluster RF=2. All 

Re: Node always dieing

[Cogumelos Maravilha]

Yes but this time I going to give lots of time between killing and pickup.

Thanks a lot.

On 04/06/2017 05:31 PM, Avi Kivity wrote:
>
> Your disk is bad.  Kill that instance and hope someone else gets it.
>
>
> On 04/06/2017 07:27 PM, Cogumelos Maravilha wrote:
>>
>> Interesting
>>
>> [  720.693768] blk_update_request: I/O error, dev nvme0n1, sector
>> 1397303056
>> [  750.698840] blk_update_request: I/O error, dev nvme0n1, sector
>> 1397303080
>> [ 1416.202103] blk_update_request: I/O error, dev nvme0n1, sector
>> 1397303080
>>
>>
>> On 04/06/2017 05:26 PM, Avi Kivity wrote:
>>>
>>> Is there anything in dmesg?
>>>
>>>
>>> On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:

 Now dies and restart (systemd) without logging why

 system.log

 INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
 AuthCache.java:172 - (Re)initializing RolesCache (validity period
 /update interval/max entries) (2000/2000/1000)
 INFO  [main] 2017-04-06 16:17:42,535
 YamlConfigurationLoader.java:89 - Configuration location:
 file:/etc/cassandra/cassandra.
 yaml


 debug.log
 DEBUG [GossipStage:1] 2017-04-06 16:16:56,272
 FailureDetector.java:457 - Ignoring interval time of 2496703934 for
 /10.100.120.52
 DEBUG [GossipStage:1] 2017-04-06 16:16:59,090
 FailureDetector.java:457 - Ignoring interval time of 2818071981 for
 /10.100.120.161
 INFO  [main] 2017-04-06 16:17:42,535
 YamlConfigurationLoader.java:89 - Configuration location:
 file:/etc/cassandra/cassandra.yaml
 DEBUG [main] 2017-04-06 16:17:42,540
 YamlConfigurationLoader.java:108 - Loading settings from
 file:/etc/cassandra/cassandra.yaml


 On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
> find /mnt/cassandra/ \! -user cassandra
> nothing
>
> I've found some "strange" solutions on Internet
> chmod -R 2777 /tmp
> chmod -R 2775 cassandra folder
>
> Lets give some time to see the result
>
> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>> All it takes is one frustrated `sudo cassandra` run. Checking only the
>> top level directory ownership is insufficient, since root could own
>> files/dirs created below the top level. Find all files not owned by user
>> cassandra:  `find /mnt/cassandra/ \! -user cassandra`
>>
>> Just another thought.
>>
>> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>>> From cassandra.yaml:
>>>
>>> hints_directory: /mnt/cassandra/hints
>>> data_file_directories:
>>> - /mnt/cassandra/data
>>> commitlog_directory: /mnt/cassandra/commitlog
>>> saved_caches_directory: /mnt/cassandra/saved_caches
>>>
>>> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>>
>>> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>>> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>>> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>>> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>>> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>>> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>>
>>> cassand+  2267 1 99 10:18 ?00:02:56 java
>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities 
>>> -XX:Threa...
>>>
>>> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>>
>>> On /etc/security/limits.conf
>>>
>>> *   -   memlock  unlimited
>>> *   -  nofile  10
>>> *   -  nproc  32768
>>> *   -  as   unlimited
>>>
>>> On /etc/security/limits.d/cassandra.conf
>>>
>>> cassandra  -  memlock  unlimited
>>> cassandra  -  nofile   10
>>> cassandra  -  as   unlimited
>>> cassandra  -  nproc32768
>>>
>>> On /etc/sysctl.conf
>>>
>>> vm.max_map_count = 1048575
>>>
>>> On /etc/systcl.d/cassanda.conf
>>>
>>> vm.max_map_count = 1048575
>>> net.ipv4.tcp_keepalive_time=600
>>>
>>> On /etc/pam.d/su
>>> ...
>>> sessionrequired   pam_limits.so
>>> ...
>>>
>>> Distro is the currently Ubuntu LTS.
>>> Thanks
>>>
>>>
>>> On 04/06/2017 10:39 AM, benjamin roth wrote:
 Cassandra cannot write an SSTable to disk. Are you sure the
 disk/volume where SSTables reside (normally /var/lib/cassandra/data)
 is writeable for the CS user and has enough free space?
 The CDC warning also implies that.
 The other warnings indicate you are probably not running CS as root
 and you did not set an appropriate limit for max open files. Running
 out of open files can also be a reason for the IO error.

 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
 >:

 

Re: Node always dieing

[Avi Kivity]

Your disk is bad.  Kill that instance and hope someone else gets it.


On 04/06/2017 07:27 PM, Cogumelos Maravilha wrote:


Interesting

[  720.693768] blk_update_request: I/O error, dev nvme0n1, sector 
1397303056
[  750.698840] blk_update_request: I/O error, dev nvme0n1, sector 
1397303080
[ 1416.202103] blk_update_request: I/O error, dev nvme0n1, sector 
1397303080



On 04/06/2017 05:26 PM, Avi Kivity wrote:


Is there anything in dmesg?


On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:


Now dies and restart (systemd) without logging why

system.log

INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362 
AuthCache.java:172 - (Re)initializing RolesCache (validity period

/update interval/max entries) (2000/2000/1000)
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 
- Configuration location: file:/etc/cassandra/cassandra.

yaml


debug.log
DEBUG [GossipStage:1] 2017-04-06 16:16:56,272 
FailureDetector.java:457 - Ignoring interval time of 2496703934 for 
/10.100.120.52
DEBUG [GossipStage:1] 2017-04-06 16:16:59,090 
FailureDetector.java:457 - Ignoring interval time of 2818071981 for 
/10.100.120.161
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 
- Configuration location: file:/etc/cassandra/cassandra.yaml
DEBUG [main] 2017-04-06 16:17:42,540 
YamlConfigurationLoader.java:108 - Loading settings from 
file:/etc/cassandra/cassandra.yaml



On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:

find/mnt/cassandra/  \! -user cassandra
nothing

I've found some "strange" solutions on Internet
chmod -R 2777 /tmp
chmod -R 2775 cassandra folder

Lets give some time to see the result

On 04/06/2017 03:14 PM, Michael Shuler wrote:

All it takes is one frustrated `sudo cassandra` run. Checking only the
top level directory ownership is insufficient, since root could own
files/dirs created below the top level. Find all files not owned by user
cassandra:  `find/mnt/cassandra/  \! -user cassandra`

Just another thought.

-- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:

 From cassandra.yaml:

hints_directory: /mnt/cassandra/hints
data_file_directories:
 - /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
saved_caches_directory: /mnt/cassandra/saved_caches

drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/

drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/

cassand+  2267 1 99 10:18 ?00:02:56 java
-Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

/dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt

On /etc/security/limits.conf

*   -   memlock  unlimited
*   -  nofile  10
*   -  nproc  32768
*   -  as   unlimited

On /etc/security/limits.d/cassandra.conf

cassandra  -  memlock  unlimited
cassandra  -  nofile   10
cassandra  -  as   unlimited
cassandra  -  nproc32768

On /etc/sysctl.conf

vm.max_map_count = 1048575

On /etc/systcl.d/cassanda.conf

vm.max_map_count = 1048575
net.ipv4.tcp_keepalive_time=600

On /etc/pam.d/su
...
sessionrequired   pam_limits.so
...

Distro is the currently Ubuntu LTS.
Thanks


On 04/06/2017 10:39 AM, benjamin roth wrote:

Cassandra cannot write an SSTable to disk. Are you sure the
disk/volume where SSTables reside (normally /var/lib/cassandra/data)
is writeable for the CS user and has enough free space?
The CDC warning also implies that.
The other warnings indicate you are probably not running CS as root
and you did not set an appropriate limit for max open files. Running
out of open files can also be a reason for the IO error.

2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>:

 Hi list,

 I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
 i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
 I have
 one node that is always dieing and I don't understand why. Can anyone
 give me some hints please. All nodes using the same configuration.

 Thanks in advance.

 INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
 IndexSummaryRedistribution.java:75 - Redistributing index summaries
 ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
 CassandraDaemon.java:229 - Exception in thread
 Thread[MemtablePostFlush:22,5,main]
 org.apache.cassandra.io
 .FSWriteError:
 java.io.IOException: Input/output
 error
 at
 
org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
 ~[apache-cassandra-3.10.jar:3.10]
 at
 

Re: Node always dieing

[Cogumelos Maravilha]

Interesting

[  720.693768] blk_update_request: I/O error, dev nvme0n1, sector 1397303056
[  750.698840] blk_update_request: I/O error, dev nvme0n1, sector 1397303080
[ 1416.202103] blk_update_request: I/O error, dev nvme0n1, sector 1397303080


On 04/06/2017 05:26 PM, Avi Kivity wrote:
>
> Is there anything in dmesg?
>
>
> On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:
>>
>> Now dies and restart (systemd) without logging why
>>
>> system.log
>>
>> INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
>> AuthCache.java:172 - (Re)initializing RolesCache (validity period
>> /update interval/max entries) (2000/2000/1000)
>> INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
>> - Configuration location: file:/etc/cassandra/cassandra.
>> yaml
>>
>>
>> debug.log
>> DEBUG [GossipStage:1] 2017-04-06 16:16:56,272
>> FailureDetector.java:457 - Ignoring interval time of 2496703934 for
>> /10.100.120.52
>> DEBUG [GossipStage:1] 2017-04-06 16:16:59,090
>> FailureDetector.java:457 - Ignoring interval time of 2818071981 for
>> /10.100.120.161
>> INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89
>> - Configuration location: file:/etc/cassandra/cassandra.yaml
>> DEBUG [main] 2017-04-06 16:17:42,540 YamlConfigurationLoader.java:108
>> - Loading settings from file:/etc/cassandra/cassandra.yaml
>>
>>
>> On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
>>> find /mnt/cassandra/ \! -user cassandra
>>> nothing
>>>
>>> I've found some "strange" solutions on Internet
>>> chmod -R 2777 /tmp
>>> chmod -R 2775 cassandra folder
>>>
>>> Lets give some time to see the result
>>>
>>> On 04/06/2017 03:14 PM, Michael Shuler wrote:
 All it takes is one frustrated `sudo cassandra` run. Checking only the
 top level directory ownership is insufficient, since root could own
 files/dirs created below the top level. Find all files not owned by user
 cassandra:  `find /mnt/cassandra/ \! -user cassandra`

 Just another thought.

 -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
> From cassandra.yaml:
>
> hints_directory: /mnt/cassandra/hints
> data_file_directories:
> - /mnt/cassandra/data
> commitlog_directory: /mnt/cassandra/commitlog
> saved_caches_directory: /mnt/cassandra/saved_caches
>
> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>
> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities 
> -XX:Threa...
>
> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>
> On /etc/security/limits.conf
>
> *   -   memlock  unlimited
> *   -  nofile  10
> *   -  nproc  32768
> *   -  as   unlimited
>
> On /etc/security/limits.d/cassandra.conf
>
> cassandra  -  memlock  unlimited
> cassandra  -  nofile   10
> cassandra  -  as   unlimited
> cassandra  -  nproc32768
>
> On /etc/sysctl.conf
>
> vm.max_map_count = 1048575
>
> On /etc/systcl.d/cassanda.conf
>
> vm.max_map_count = 1048575
> net.ipv4.tcp_keepalive_time=600
>
> On /etc/pam.d/su
> ...
> sessionrequired   pam_limits.so
> ...
>
> Distro is the currently Ubuntu LTS.
> Thanks
>
>
> On 04/06/2017 10:39 AM, benjamin roth wrote:
>> Cassandra cannot write an SSTable to disk. Are you sure the
>> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
>> is writeable for the CS user and has enough free space?
>> The CDC warning also implies that.
>> The other warnings indicate you are probably not running CS as root
>> and you did not set an appropriate limit for max open files. Running
>> out of open files can also be a reason for the IO error.
>>
>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>> >:
>>
>> Hi list,
>>
>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
>> I have
>> one node that is always dieing and I don't understand why. Can anyone
>> give me some hints please. All nodes using the same configuration.
>>
>> Thanks in advance.
>>
>> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>> IndexSummaryRedistribution.java:75 - Redistributing index summaries
>> 

Re: Node always dieing

[Avi Kivity]

Is there anything in dmesg?


On 04/06/2017 07:25 PM, Cogumelos Maravilha wrote:


Now dies and restart (systemd) without logging why

system.log

INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362 
AuthCache.java:172 - (Re)initializing RolesCache (validity period

/update interval/max entries) (2000/2000/1000)
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 - 
Configuration location: file:/etc/cassandra/cassandra.

yaml


debug.log
DEBUG [GossipStage:1] 2017-04-06 16:16:56,272 FailureDetector.java:457 
- Ignoring interval time of 2496703934 for /10.100.120.52
DEBUG [GossipStage:1] 2017-04-06 16:16:59,090 FailureDetector.java:457 
- Ignoring interval time of 2818071981 for /10.100.120.161
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 - 
Configuration location: file:/etc/cassandra/cassandra.yaml
DEBUG [main] 2017-04-06 16:17:42,540 YamlConfigurationLoader.java:108 
- Loading settings from file:/etc/cassandra/cassandra.yaml



On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:

find/mnt/cassandra/  \! -user cassandra
nothing

I've found some "strange" solutions on Internet
chmod -R 2777 /tmp
chmod -R 2775 cassandra folder

Lets give some time to see the result

On 04/06/2017 03:14 PM, Michael Shuler wrote:

All it takes is one frustrated `sudo cassandra` run. Checking only the
top level directory ownership is insufficient, since root could own
files/dirs created below the top level. Find all files not owned by user
cassandra:  `find/mnt/cassandra/  \! -user cassandra`

Just another thought.

-- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:

 From cassandra.yaml:

hints_directory: /mnt/cassandra/hints
data_file_directories:
 - /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
saved_caches_directory: /mnt/cassandra/saved_caches

drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/

drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/

cassand+  2267 1 99 10:18 ?00:02:56 java
-Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

/dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt

On /etc/security/limits.conf

*   -   memlock  unlimited
*   -  nofile  10
*   -  nproc  32768
*   -  as   unlimited

On /etc/security/limits.d/cassandra.conf

cassandra  -  memlock  unlimited
cassandra  -  nofile   10
cassandra  -  as   unlimited
cassandra  -  nproc32768

On /etc/sysctl.conf

vm.max_map_count = 1048575

On /etc/systcl.d/cassanda.conf

vm.max_map_count = 1048575
net.ipv4.tcp_keepalive_time=600

On /etc/pam.d/su
...
sessionrequired   pam_limits.so
...

Distro is the currently Ubuntu LTS.
Thanks


On 04/06/2017 10:39 AM, benjamin roth wrote:

Cassandra cannot write an SSTable to disk. Are you sure the
disk/volume where SSTables reside (normally /var/lib/cassandra/data)
is writeable for the CS user and has enough free space?
The CDC warning also implies that.
The other warnings indicate you are probably not running CS as root
and you did not set an appropriate limit for max open files. Running
out of open files can also be a reason for the IO error.

2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>:

 Hi list,

 I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
 i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
 I have
 one node that is always dieing and I don't understand why. Can anyone
 give me some hints please. All nodes using the same configuration.

 Thanks in advance.

 INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
 IndexSummaryRedistribution.java:75 - Redistributing index summaries
 ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
 CassandraDaemon.java:229 - Exception in thread
 Thread[MemtablePostFlush:22,5,main]
 org.apache.cassandra.io
 .FSWriteError:
 java.io.IOException: Input/output
 error
 at
 
org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
 ~[apache-cassandra-3.10.jar:3.10]
 at
 
org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
 ~[apache-cassandra-3.10.jar:3.10]
 at
 org.apache.cassandra.io
 
.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
 ~[apache-cassandra-3.10.jar:3.10]
 at
 org.apache.cassandra.io
 

Re: Node always dieing

[Cogumelos Maravilha]

Now dies and restart (systemd) without logging why

system.log

INFO  [Native-Transport-Requests-2] 2017-04-06 16:06:55,362
AuthCache.java:172 - (Re)initializing RolesCache (validity period
/update interval/max entries) (2000/2000/1000)
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 -
Configuration location: file:/etc/cassandra/cassandra.
yaml


debug.log
DEBUG [GossipStage:1] 2017-04-06 16:16:56,272 FailureDetector.java:457 -
Ignoring interval time of 2496703934 for /10.100.120.52
DEBUG [GossipStage:1] 2017-04-06 16:16:59,090 FailureDetector.java:457 -
Ignoring interval time of 2818071981 for /10.100.120.161
INFO  [main] 2017-04-06 16:17:42,535 YamlConfigurationLoader.java:89 -
Configuration location: file:/etc/cassandra/cassandra.yaml
DEBUG [main] 2017-04-06 16:17:42,540 YamlConfigurationLoader.java:108 -
Loading settings from file:/etc/cassandra/cassandra.yaml


On 04/06/2017 04:18 PM, Cogumelos Maravilha wrote:
> find /mnt/cassandra/ \! -user cassandra
> nothing
>
> I've found some "strange" solutions on Internet
> chmod -R 2777 /tmp
> chmod -R 2775 cassandra folder
>
> Lets give some time to see the result
>
> On 04/06/2017 03:14 PM, Michael Shuler wrote:
>> All it takes is one frustrated `sudo cassandra` run. Checking only the
>> top level directory ownership is insufficient, since root could own
>> files/dirs created below the top level. Find all files not owned by user
>> cassandra:  `find /mnt/cassandra/ \! -user cassandra`
>>
>> Just another thought.
>>
>> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>>> From cassandra.yaml:
>>>
>>> hints_directory: /mnt/cassandra/hints
>>> data_file_directories:
>>> - /mnt/cassandra/data
>>> commitlog_directory: /mnt/cassandra/commitlog
>>> saved_caches_directory: /mnt/cassandra/saved_caches
>>>
>>> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>>
>>> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>>> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>>> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>>> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>>> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>>> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>>
>>> cassand+  2267 1 99 10:18 ?00:02:56 java
>>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
>>>
>>> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>>
>>> On /etc/security/limits.conf
>>>
>>> *   -   memlock  unlimited
>>> *   -  nofile  10
>>> *   -  nproc  32768
>>> *   -  as   unlimited
>>>
>>> On /etc/security/limits.d/cassandra.conf
>>>
>>> cassandra  -  memlock  unlimited
>>> cassandra  -  nofile   10
>>> cassandra  -  as   unlimited
>>> cassandra  -  nproc32768
>>>
>>> On /etc/sysctl.conf
>>>
>>> vm.max_map_count = 1048575
>>>
>>> On /etc/systcl.d/cassanda.conf
>>>
>>> vm.max_map_count = 1048575
>>> net.ipv4.tcp_keepalive_time=600
>>>
>>> On /etc/pam.d/su
>>> ...
>>> sessionrequired   pam_limits.so
>>> ...
>>>
>>> Distro is the currently Ubuntu LTS.
>>> Thanks
>>>
>>>
>>> On 04/06/2017 10:39 AM, benjamin roth wrote:
 Cassandra cannot write an SSTable to disk. Are you sure the
 disk/volume where SSTables reside (normally /var/lib/cassandra/data)
 is writeable for the CS user and has enough free space?
 The CDC warning also implies that.
 The other warnings indicate you are probably not running CS as root
 and you did not set an appropriate limit for max open files. Running
 out of open files can also be a reason for the IO error.

 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
 >:

 Hi list,

 I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
 i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
 I have
 one node that is always dieing and I don't understand why. Can anyone
 give me some hints please. All nodes using the same configuration.

 Thanks in advance.

 INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
 IndexSummaryRedistribution.java:75 - Redistributing index summaries
 ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
 CassandraDaemon.java:229 - Exception in thread
 Thread[MemtablePostFlush:22,5,main]
 org.apache.cassandra.io
 .FSWriteError:
 java.io.IOException: Input/output
 error
 at
 
 org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
 ~[apache-cassandra-3.10.jar:3.10]
 at
 
 org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
 ~[apache-cassandra-3.10.jar:3.10]
 at
 

Re: Node always dieing

[Cogumelos Maravilha]

find /mnt/cassandra/ \! -user cassandra
nothing

I've found some "strange" solutions on Internet
chmod -R 2777 /tmp
chmod -R 2775 cassandra folder

Lets give some time to see the result


On 04/06/2017 03:14 PM, Michael Shuler wrote:
> All it takes is one frustrated `sudo cassandra` run. Checking only the
> top level directory ownership is insufficient, since root could own
> files/dirs created below the top level. Find all files not owned by user
> cassandra:  `find /mnt/cassandra/ \! -user cassandra`
>
> Just another thought.
>
> -- Michael On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
>> From cassandra.yaml:
>>
>> hints_directory: /mnt/cassandra/hints
>> data_file_directories:
>> - /mnt/cassandra/data
>> commitlog_directory: /mnt/cassandra/commitlog
>> saved_caches_directory: /mnt/cassandra/saved_caches
>>
>> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>
>> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>
>> cassand+  2267 1 99 10:18 ?00:02:56 java
>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
>>
>> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>
>> On /etc/security/limits.conf
>>
>> *   -   memlock  unlimited
>> *   -  nofile  10
>> *   -  nproc  32768
>> *   -  as   unlimited
>>
>> On /etc/security/limits.d/cassandra.conf
>>
>> cassandra  -  memlock  unlimited
>> cassandra  -  nofile   10
>> cassandra  -  as   unlimited
>> cassandra  -  nproc32768
>>
>> On /etc/sysctl.conf
>>
>> vm.max_map_count = 1048575
>>
>> On /etc/systcl.d/cassanda.conf
>>
>> vm.max_map_count = 1048575
>> net.ipv4.tcp_keepalive_time=600
>>
>> On /etc/pam.d/su
>> ...
>> sessionrequired   pam_limits.so
>> ...
>>
>> Distro is the currently Ubuntu LTS.
>> Thanks
>>
>>
>> On 04/06/2017 10:39 AM, benjamin roth wrote:
>>> Cassandra cannot write an SSTable to disk. Are you sure the
>>> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
>>> is writeable for the CS user and has enough free space?
>>> The CDC warning also implies that.
>>> The other warnings indicate you are probably not running CS as root
>>> and you did not set an appropriate limit for max open files. Running
>>> out of open files can also be a reason for the IO error.
>>>
>>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>>> >:
>>>
>>> Hi list,
>>>
>>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
>>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
>>> I have
>>> one node that is always dieing and I don't understand why. Can anyone
>>> give me some hints please. All nodes using the same configuration.
>>>
>>> Thanks in advance.
>>>
>>> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>>> IndexSummaryRedistribution.java:75 - Redistributing index summaries
>>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>> CassandraDaemon.java:229 - Exception in thread
>>> Thread[MemtablePostFlush:22,5,main]
>>> org.apache.cassandra.io
>>> .FSWriteError:
>>> java.io.IOException: Input/output
>>> error
>>> at
>>> 
>>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> 
>>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io
>>> 
>>> .compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io
>>> 
>>> .compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> 
>>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> 
>>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> org.apache.cassandra.io
>>> 
>>> .sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
>>> ~[apache-cassandra-3.10.jar:3.10]
>>> at
>>> 
>>> 

Re: Node always dieing

[Michael Shuler]

All it takes is one frustrated `sudo cassandra` run. Checking only the
top level directory ownership is insufficient, since root could own
files/dirs created below the top level. Find all files not owned by user
cassandra:  `find /mnt/cassandra/ \! -user cassandra`

Just another thought.

-- 
Michael


On 04/06/2017 05:23 AM, Cogumelos Maravilha wrote:
> From cassandra.yaml:
> 
> hints_directory: /mnt/cassandra/hints
> data_file_directories:
> - /mnt/cassandra/data
> commitlog_directory: /mnt/cassandra/commitlog
> saved_caches_directory: /mnt/cassandra/saved_caches
> 
> drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
> 
> drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
> drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
> drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
> drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
> drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
> drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
> 
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...
> 
> /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
> 
> On /etc/security/limits.conf
> 
> *   -   memlock  unlimited
> *   -  nofile  10
> *   -  nproc  32768
> *   -  as   unlimited
> 
> On /etc/security/limits.d/cassandra.conf
> 
> cassandra  -  memlock  unlimited
> cassandra  -  nofile   10
> cassandra  -  as   unlimited
> cassandra  -  nproc32768
> 
> On /etc/sysctl.conf
> 
> vm.max_map_count = 1048575
> 
> On /etc/systcl.d/cassanda.conf
> 
> vm.max_map_count = 1048575
> net.ipv4.tcp_keepalive_time=600
> 
> On /etc/pam.d/su
> ...
> sessionrequired   pam_limits.so
> ...
> 
> Distro is the currently Ubuntu LTS.
> Thanks
> 
> 
> On 04/06/2017 10:39 AM, benjamin roth wrote:
>> Cassandra cannot write an SSTable to disk. Are you sure the
>> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
>> is writeable for the CS user and has enough free space?
>> The CDC warning also implies that.
>> The other warnings indicate you are probably not running CS as root
>> and you did not set an appropriate limit for max open files. Running
>> out of open files can also be a reason for the IO error.
>>
>> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>> >:
>>
>> Hi list,
>>
>> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
>> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
>> I have
>> one node that is always dieing and I don't understand why. Can anyone
>> give me some hints please. All nodes using the same configuration.
>>
>> Thanks in advance.
>>
>> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>> IndexSummaryRedistribution.java:75 - Redistributing index summaries
>> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>> CassandraDaemon.java:229 - Exception in thread
>> Thread[MemtablePostFlush:22,5,main]
>> org.apache.cassandra.io
>> .FSWriteError:
>> java.io.IOException: Input/output
>> error
>> at
>> 
>> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> 
>> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> org.apache.cassandra.io
>> 
>> .compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> org.apache.cassandra.io
>> 
>> .compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> 
>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> 
>> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> org.apache.cassandra.io
>> 
>> .sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> 
>> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> org.apache.cassandra.io
>> 
>> .sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
>> ~[apache-cassandra-3.10.jar:3.10]
>> at
>> 

Re: Node always dieing

[Cogumelos Maravilha]

We tested in c4 instances but EBS is too slow. So we deployed for
production in i3.

It was running with 5 nodes without problems but we started running out
of space so we added another node. And is this last node that is giving
problems. I've already terminated the instance and created another once,
but the problem remains. The configuration is the same for all
instances. Runs nice for a few time but then

ERROR [MemtablePostFlush:11] 2017-04-06 11:47:40,840
CassandraDaemon.java:229 - Exception in thread
Thread[MemtablePostFlush:11,5,main]

Thanks.

On 04/06/2017 01:18 PM, Carlos Rolo wrote:
> There was some issue with the i3 instances and Cassandra. Did you had
> this cluster running always on i3? 
>
> On Apr 6, 2017 13:06, "Cogumelos Maravilha"
> > wrote:
>
> Limit Soft Limit   Hard
> Limit   Units
> Max cpu time  unlimited   
> unlimitedseconds  
> Max file size unlimited   
> unlimitedbytes
> Max data size unlimited   
> unlimitedbytes
> Max stack size8388608 
> unlimitedbytes
> Max core file size0   
> unlimitedbytes
> Max resident set  unlimited   
> unlimitedbytes
> Max processes 122575  
> 122575   processes
> Max open files10  
> 10   files
> Max locked memory unlimited   
> unlimitedbytes
> Max address space unlimited   
> unlimitedbytes
> Max file locksunlimited   
> unlimitedlocks
> Max pending signals   122575  
> 122575   signals  
> Max msgqueue size 819200  
> 819200   bytes
> Max nice priority 00   
> Max realtime priority 00   
> Max realtime timeout  unlimited   
> unlimitedus   
>
> Please find something wrong there!
>
> Thanks.
>
> On 04/06/2017 11:50 AM, benjamin roth wrote:
>> Limits: You should check them in /proc/$pid/limits
>>
>> 2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha
>> >:
>>
>> Yes C* is running as cassandra:
>>
>> cassand+  2267 1 99 10:18 ?00:02:56 java
>> -Xloggc:/var/log/cassandra/gc.log -ea
>> -XX:+UseThreadPriorities -XX:Threa...
>>
>> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>> configuration:[allocate_tokens_for_keyspace=null;
>> authenticator=PasswordAuthenticator;
>> authorizer=CassandraAuthorizer; auto_bootstrap=true;
>> auto_snapshot=true; back_pressure_enabled=false;
>> back_pressure_strategy=org.apache.cassandra.net
>> .RateBasedBackPressure{high_ratio=0.9,
>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>> batch_size_warn_threshold_in_kb=5;
>> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
>> broadcast_rpc_address=null;
>> buffer_pool_use_heap_if_exhausted=true;
>> cas_contention_timeout_in_ms=600; cdc_enabled=false;
>> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
>> cdc_total_space_in_mb=0;
>> client_encryption_options=; cluster_name=company;
>> column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
>> commit_failure_policy=ignore; commitlog_compression=null;
>> commitlog_directory=/mnt/cassandra/commitlog;
>> commitlog_max_compression_buffers_in_pool=3;
>> commitlog_periodic_queue_size=-1;
>> commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
>> commitlog_sync_batch_window_in_ms=NaN;
>> commitlog_sync_period_in_ms=1;
>> commitlog_total_space_in_mb=null;
>> compaction_large_partition_warning_threshold_mb=100;
>> compaction_throughput_mb_per_sec=16;
>> concurrent_compactors=null; concurrent_counter_writes=32;
>> concurrent_materialized_view_writes=32; concurrent_reads=32;
>> concurrent_replicates=null; concurrent_writes=32;
>> counter_cache_keys_to_save=2147483647;
>> counter_cache_save_period=7200;
>> counter_cache_size_in_mb=null;
>> counter_write_request_timeout_in_ms=600;
>> credentials_cache_max_entries=1000;
>> credentials_update_interval_in_ms=-1;
>> credentials_validity_in_ms=2000; cross_node_timeout=false;
>> 

Re: Node always dieing

[Carlos Rolo]

There was some issue with the i3 instances and Cassandra. Did you had this
cluster running always on i3?

On Apr 6, 2017 13:06, "Cogumelos Maravilha" 
wrote:

> Limit Soft Limit   Hard Limit
> Units
> Max cpu time  unlimitedunlimited
> seconds
> Max file size unlimitedunlimited
> bytes
> Max data size unlimitedunlimited
> bytes
> Max stack size8388608  unlimited
> bytes
> Max core file size0unlimited
> bytes
> Max resident set  unlimitedunlimited
> bytes
> Max processes 122575   122575
> processes
> Max open files10   10
> files
> Max locked memory unlimitedunlimited
> bytes
> Max address space unlimitedunlimited
> bytes
> Max file locksunlimitedunlimited
> locks
> Max pending signals   122575   122575
> signals
> Max msgqueue size 819200   819200
> bytes
> Max nice priority 00
> Max realtime priority 00
> Max realtime timeout  unlimitedunlimitedus
> Please find something wrong there!
>
> Thanks.
>
> On 04/06/2017 11:50 AM, benjamin roth wrote:
>
> Limits: You should check them in /proc/$pid/limits
>
> 2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha  >:
>
>> Yes C* is running as cassandra:
>>
>> cassand+  2267 1 99 10:18 ?00:02:56 java
>> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
>> -XX:Threa...
>>
>> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>> configuration:[allocate_tokens_for_keyspace=null;
>> authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer;
>> auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false;
>> back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>> broadcast_address=null; broadcast_rpc_address=null;
>> buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=600;
>> cdc_enabled=false; cdc_free_space_check_interval_ms=250;
>> cdc_raw_directory=null; cdc_total_space_in_mb=0;
>> client_encryption_options=; cluster_name=company;
>> column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
>> commit_failure_policy=ignore; commitlog_compression=null;
>> commitlog_directory=/mnt/cassandra/commitlog;
>> commitlog_max_compression_buffers_in_pool=3;
>> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
>> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
>> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=null;
>> compaction_large_partition_warning_threshold_mb=100;
>> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
>> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
>> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32;
>> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=600;
>> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
>> credentials_validity_in_ms=2000; cross_node_timeout=false;
>> data_file_directories=[Ljava.lang.String;@223f3642;
>> disk_access_mode=auto; disk_failure_policy=ignore;
>> disk_optimization_estimate_percentile=0.95;
>> disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
>> dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
>> dynamic_snitch_reset_interval_in_ms=60;
>> dynamic_snitch_update_interval_in_ms=100; 
>> enable_scripted_user_defined_functions=false;
>> enable_user_defined_functions=false; 
>> enable_user_defined_functions_threads=true;
>> encryption_options=null; endpoint_snitch=SimpleSnitch;
>> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
>> gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[];
>> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
>> hints_compression=null; hints_directory=/mnt/cassandra/hints;
>> hints_flush_period_in_ms=1; incremental_backups=false;
>> index_interval=null; index_summary_capacity_in_mb=null;
>> index_summary_resize_interval_in_minutes=60; initial_token=null;
>> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
>> inter_dc_tcp_nodelay=false; internode_authenticator=null;
>> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
>> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
>> key_cache_save_period=14400; key_cache_size_in_mb=null;
>> listen_address=10.100.100.213; listen_interface=null;
>> listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
>> 

Re: Node always dieing

[Cogumelos Maravilha]

Limit Soft Limit   Hard Limit  
Units
Max cpu time  unlimitedunlimited   
seconds  
Max file size unlimitedunlimited   
bytes
Max data size unlimitedunlimited   
bytes
Max stack size8388608  unlimited   
bytes
Max core file size0unlimited   
bytes
Max resident set  unlimitedunlimited   
bytes
Max processes 122575   122575  
processes
Max open files10   10  
files
Max locked memory unlimitedunlimited   
bytes
Max address space unlimitedunlimited   
bytes
Max file locksunlimitedunlimited   
locks
Max pending signals   122575   122575  
signals  
Max msgqueue size 819200   819200  
bytes
Max nice priority 00   
Max realtime priority 00   
Max realtime timeout  unlimitedunlimitedus   

Please find something wrong there!

Thanks.

On 04/06/2017 11:50 AM, benjamin roth wrote:
> Limits: You should check them in /proc/$pid/limits
>
> 2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha
> >:
>
> Yes C* is running as cassandra:
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
> -XX:Threa...
>
> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
> configuration:[allocate_tokens_for_keyspace=null;
> authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_bootstrap=true;
> auto_snapshot=true; back_pressure_enabled=false;
> back_pressure_strategy=org.apache.cassandra.net
> .RateBasedBackPressure{high_ratio=0.9,
> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5;
> batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
> broadcast_rpc_address=null;
> buffer_pool_use_heap_if_exhausted=true;
> cas_contention_timeout_in_ms=600; cdc_enabled=false;
> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
> cdc_total_space_in_mb=0; client_encryption_options=;
> cluster_name=company; column_index_cache_size_in_kb=2;
> column_index_size_in_kb=64; commit_failure_policy=ignore;
> commitlog_compression=null;
> commitlog_directory=/mnt/cassandra/commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1;
> commitlog_total_space_in_mb=null;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
> concurrent_counter_writes=32;
> concurrent_materialized_view_writes=32; concurrent_reads=32;
> concurrent_replicates=null; concurrent_writes=32;
> counter_cache_keys_to_save=2147483647;
> counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> counter_write_request_timeout_in_ms=600;
> credentials_cache_max_entries=1000;
> credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@223f3642;
> disk_access_mode=auto; disk_failure_policy=ignore;
> disk_optimization_estimate_percentile=0.95;
> disk_optimization_page_cross_chance=0.1;
> disk_optimization_strategy=ssd; dynamic_snitch=true;
> dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100;
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false;
> enable_user_defined_functions_threads=true;
> encryption_options=null; endpoint_snitch=SimpleSnitch;
> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
> gc_warn_threshold_in_ms=1000;
> hinted_handoff_disabled_datacenters=[];
> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> hints_compression=null; hints_directory=/mnt/cassandra/hints;
> hints_flush_period_in_ms=1; incremental_backups=false;
> index_interval=null; index_summary_capacity_in_mb=null;
> index_summary_resize_interval_in_minutes=60; initial_token=null;
> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
> inter_dc_tcp_nodelay=false; internode_authenticator=null;
> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
> 

Re: Node always dieing

[benjamin roth]

Limits: You should check them in /proc/$pid/limits

2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha :

> Yes C* is running as cassandra:
>
> cassand+  2267 1 99 10:18 ?00:02:56 java
> -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
> -XX:Threa...
>
> INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
> configuration:[allocate_tokens_for_keyspace=null; 
> authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_bootstrap=true; auto_snapshot=true;
> back_pressure_enabled=false; back_pressure_strategy=org.
> apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5,
> flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> broadcast_address=null; broadcast_rpc_address=null; 
> buffer_pool_use_heap_if_exhausted=true;
> cas_contention_timeout_in_ms=600; cdc_enabled=false;
> cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
> cdc_total_space_in_mb=0; client_encryption_options=;
> cluster_name=company; column_index_cache_size_in_kb=2;
> column_index_size_in_kb=64; commit_failure_policy=ignore;
> commitlog_compression=null; commitlog_directory=/mnt/cassandra/commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=null;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32;
> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=600;
> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@223f3642;
> disk_access_mode=auto; disk_failure_policy=ignore;
> disk_optimization_estimate_percentile=0.95; 
> disk_optimization_page_cross_chance=0.1;
> disk_optimization_strategy=ssd; dynamic_snitch=true;
> dynamic_snitch_badness_threshold=0.1; 
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100; 
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false; 
> enable_user_defined_functions_threads=true;
> encryption_options=null; endpoint_snitch=SimpleSnitch;
> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
> gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[];
> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> hints_compression=null; hints_directory=/mnt/cassandra/hints;
> hints_flush_period_in_ms=1; incremental_backups=false;
> index_interval=null; index_summary_capacity_in_mb=null;
> index_summary_resize_interval_in_minutes=60; initial_token=null;
> inter_dc_stream_throughput_outbound_megabits_per_sec=200;
> inter_dc_tcp_nodelay=false; internode_authenticator=null;
> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
> key_cache_save_period=14400; key_cache_size_in_mb=null;
> listen_address=10.100.100.213; listen_interface=null;
> listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
> max_hint_window_in_ms=1080; max_hints_delivery_threads=2;
> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
> max_streaming_retries=3; max_value_size_in_mb=256;
> memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null;
> memtable_flush_writers=0; memtable_heap_space_in_mb=null;
> memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50;
> native_transport_max_concurrent_connections=-1; native_transport_max_
> concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256;
> native_transport_max_threads=128; native_transport_port=9042;
> native_transport_port_ssl=null; num_tokens=256; 
> otc_coalescing_strategy=TIMEHORIZON;
> otc_coalescing_window_us=200; 
> partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1;
> permissions_validity_in_ms=2000; phi_convict_threshold=8.0;
> prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=600;
> read_request_timeout_in_ms=600; request_scheduler=org.apache.
> cassandra.scheduler.NoScheduler; request_scheduler_id=null;
> request_scheduler_options=null; request_timeout_in_ms=600;
> role_manager=CassandraRoleManager; roles_cache_max_entries=1000;
> roles_update_interval_in_ms=-1; roles_validity_in_ms=2000;
> row_cache_class_name=org.apache.cassandra.cache.OHCProvider;
> row_cache_keys_to_save=2147483647; row_cache_save_period=0;
> 

Re: Node always dieing

[Cogumelos Maravilha]

Yes C* is running as cassandra:

cassand+  2267 1 99 10:18 ?00:02:56 java
-Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
configuration:[allocate_tokens_for_keyspace=null;
authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer;
auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false;
back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
broadcast_address=null; broadcast_rpc_address=null;
buffer_pool_use_heap_if_exhausted=true;
cas_contention_timeout_in_ms=600; cdc_enabled=false;
cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
cdc_total_space_in_mb=0; client_encryption_options=;
cluster_name=company; column_index_cache_size_in_kb=2;
column_index_size_in_kb=64; commit_failure_policy=ignore;
commitlog_compression=null;
commitlog_directory=/mnt/cassandra/commitlog;
commitlog_max_compression_buffers_in_pool=3;
commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=null;
compaction_large_partition_warning_threshold_mb=100;
compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32;
counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
counter_cache_size_in_mb=null;
counter_write_request_timeout_in_ms=600;
credentials_cache_max_entries=1000;
credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000;
cross_node_timeout=false;
data_file_directories=[Ljava.lang.String;@223f3642;
disk_access_mode=auto; disk_failure_policy=ignore;
disk_optimization_estimate_percentile=0.95;
disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
dynamic_snitch_reset_interval_in_ms=60;
dynamic_snitch_update_interval_in_ms=100;
enable_scripted_user_defined_functions=false;
enable_user_defined_functions=false;
enable_user_defined_functions_threads=true; encryption_options=null;
endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null;
gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000;
hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true;
hinted_handoff_throttle_in_kb=1024; hints_compression=null;
hints_directory=/mnt/cassandra/hints; hints_flush_period_in_ms=1;
incremental_backups=false; index_interval=null;
index_summary_capacity_in_mb=null;
index_summary_resize_interval_in_minutes=60; initial_token=null;
inter_dc_stream_throughput_outbound_megabits_per_sec=200;
inter_dc_tcp_nodelay=false; internode_authenticator=null;
internode_compression=dc; internode_recv_buff_size_in_bytes=0;
internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
key_cache_save_period=14400; key_cache_size_in_mb=null;
listen_address=10.100.100.213; listen_interface=null;
listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
max_hint_window_in_ms=1080; max_hints_delivery_threads=2;
max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
max_streaming_retries=3; max_value_size_in_mb=256;
memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null;
memtable_flush_writers=0; memtable_heap_space_in_mb=null;
memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50;
native_transport_max_concurrent_connections=-1;
native_transport_max_concurrent_connections_per_ip=-1;
native_transport_max_frame_size_in_mb=256;
native_transport_max_threads=128; native_transport_port=9042;
native_transport_port_ssl=null; num_tokens=256;
otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200;
partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
permissions_cache_max_entries=1000;
permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000;
phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null;
range_request_timeout_in_ms=600; read_request_timeout_in_ms=600;
request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
request_scheduler_id=null; request_scheduler_options=null;
request_timeout_in_ms=600; role_manager=CassandraRoleManager;
roles_cache_max_entries=1000; roles_update_interval_in_ms=-1;
roles_validity_in_ms=2000;
row_cache_class_name=org.apache.cassandra.cache.OHCProvider;
row_cache_keys_to_save=2147483647; row_cache_save_period=0;
row_cache_size_in_mb=0; rpc_address=10.100.100.213; rpc_interface=null;
rpc_interface_prefer_ipv6=false; rpc_keepalive=true;
rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16;
rpc_port=9160; rpc_recv_buff_size_in_bytes=null;
rpc_send_buff_size_in_bytes=null; rpc_server_type=sync;

Re: Node always dieing

[benjamin roth]

Have you checked the effective limits of a running CS process?
Is CS run as Cassandra? Just to rule out missing file perms.


Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha" <
cogumelosmaravi...@sapo.pt>:

>From cassandra.yaml:
hints_directory: /mnt/cassandra/hints
data_file_directories:
- /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
saved_caches_directory: /mnt/cassandra/saved_caches

drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/

drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/

cassand+  2267 1 99 10:18 ?00:02:56 java
-Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

/dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt

On /etc/security/limits.conf

*   -   memlock  unlimited
*   -  nofile  10
*   -  nproc  32768
*   -  as   unlimited

On /etc/security/limits.d/cassandra.conf

cassandra  -  memlock  unlimited
cassandra  -  nofile   10
cassandra  -  as   unlimited
cassandra  -  nproc32768

On /etc/sysctl.conf

vm.max_map_count = 1048575

On /etc/systcl.d/cassanda.conf

vm.max_map_count = 1048575
net.ipv4.tcp_keepalive_time=600
On /etc/pam.d/su
...
sessionrequired   pam_limits.so
...

Distro is the currently Ubuntu LTS.
Thanks



On 04/06/2017 10:39 AM, benjamin roth wrote:

Cassandra cannot write an SSTable to disk. Are you sure the disk/volume
where SSTables reside (normally /var/lib/cassandra/data) is writeable for
the CS user and has enough free space?
The CDC warning also implies that.
The other warnings indicate you are probably not running CS as root and you
did not set an appropriate limit for max open files. Running out of open
files can also be a reason for the IO error.

2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha :

> Hi list,
>
> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G. I have
> one node that is always dieing and I don't understand why. Can anyone
> give me some hints please. All nodes using the same configuration.
>
> Thanks in advance.
>
> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
> IndexSummaryRedistribution.java:75 - Redistributing index summaries
> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
> CassandraDaemon.java:229 - Exception in thread
> Thread[MemtablePostFlush:22,5,main]
> org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
> error
> at
> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn
> ternal(SequentialWriter.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.util.SequentialWriter.syncInternal(S
> equentialWriter.java:185)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.
> access$100(CompressedSequentialWriter.java:38)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter$
> TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$Abstract
> Transactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.util.SequentialWriter.prepareToCommi
> t(SequentialWriter.java:358)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$Tr
> ansactionalProxy.doPrepare(BigTableWriter.java:367)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$Abstract
> Transactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepare
> ToCommit(SSTableWriter.java:281)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.pre
> pareToCommit(SimpleSSTableMultiWriter.java:101)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtabl
> e(ColumnFamilyStore.java:1153)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFa
> milyStore.java:1086)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$
> threadLocalDeallocator$0(NamedThreadFactory.java:79)
> 

Re: Node always dieing

[Cogumelos Maravilha]

From cassandra.yaml:

hints_directory: /mnt/cassandra/hints
data_file_directories:
- /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
saved_caches_directory: /mnt/cassandra/saved_caches

drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/

drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/

cassand+  2267 1 99 10:18 ?00:02:56 java
-Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

/dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt

On /etc/security/limits.conf

*   -   memlock  unlimited
*   -  nofile  10
*   -  nproc  32768
*   -  as   unlimited

On /etc/security/limits.d/cassandra.conf

cassandra  -  memlock  unlimited
cassandra  -  nofile   10
cassandra  -  as   unlimited
cassandra  -  nproc32768

On /etc/sysctl.conf

vm.max_map_count = 1048575

On /etc/systcl.d/cassanda.conf

vm.max_map_count = 1048575
net.ipv4.tcp_keepalive_time=600

On /etc/pam.d/su
...
sessionrequired   pam_limits.so
...

Distro is the currently Ubuntu LTS.
Thanks


On 04/06/2017 10:39 AM, benjamin roth wrote:
> Cassandra cannot write an SSTable to disk. Are you sure the
> disk/volume where SSTables reside (normally /var/lib/cassandra/data)
> is writeable for the CS user and has enough free space?
> The CDC warning also implies that.
> The other warnings indicate you are probably not running CS as root
> and you did not set an appropriate limit for max open files. Running
> out of open files can also be a reason for the IO error.
>
> 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
> >:
>
> Hi list,
>
> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G.
> I have
> one node that is always dieing and I don't understand why. Can anyone
> give me some hints please. All nodes using the same configuration.
>
> Thanks in advance.
>
> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
> IndexSummaryRedistribution.java:75 - Redistributing index summaries
> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
> CassandraDaemon.java:229 - Exception in thread
> Thread[MemtablePostFlush:22,5,main]
> org.apache.cassandra.io
> .FSWriteError:
> java.io.IOException: Input/output
> error
> at
> 
> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> .compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> .compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> .sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> .sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io
> 
> .sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> 
> 

Re: Node always dieing

[benjamin roth]

Cassandra cannot write an SSTable to disk. Are you sure the disk/volume
where SSTables reside (normally /var/lib/cassandra/data) is writeable for
the CS user and has enough free space?
The CDC warning also implies that.
The other warnings indicate you are probably not running CS as root and you
did not set an appropriate limit for max open files. Running out of open
files can also be a reason for the IO error.

2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha :

> Hi list,
>
> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type
> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G. I have
> one node that is always dieing and I don't understand why. Can anyone
> give me some hints please. All nodes using the same configuration.
>
> Thanks in advance.
>
> INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
> IndexSummaryRedistribution.java:75 - Redistributing index summaries
> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
> CassandraDaemon.java:229 - Exception in thread
> Thread[MemtablePostFlush:22,5,main]
> org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
> error
> at
> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(
> SequentialWriter.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.util.SequentialWriter.syncInternal(
> SequentialWriter.java:185)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.access$100(
> CompressedSequentialWriter.java:38)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter$
> TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.
> prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(
> SequentialWriter.java:358)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$
> TransactionalProxy.doPrepare(BigTableWriter.java:367)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.
> prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.
> prepareToCommit(SSTableWriter.java:281)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.prepareToCommit(
> SimpleSSTableMultiWriter.java:101)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(
> ColumnFamilyStore.java:1153)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(
> ColumnFamilyStore.java:1086)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.10.jar:3.10]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> Caused by: java.io.IOException: Input/output error
> at sun.nio.ch.FileDispatcherImpl.force0(Native Method) ~[na:1.8.0_121]
> at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
> ~[na:1.8.0_121]
> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
> ~[na:1.8.0_121]
> at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(
> SequentialWriter.java:169)
> ~[apache-cassandra-3.10.jar:3.10]
> ... 15 common frames omitted
> INFO  [IndexSummaryManager:1] 2017-04-06 06:22:18,366
> IndexSummaryRedistribution.java:75 - Redistributing index summaries
> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
> CassandraDaemon.java:229 - Exception in thread
> Thread[MemtablePostFlush:31,5,main]
> org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
> error
> at
> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(
> SequentialWriter.java:173)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.util.SequentialWriter.syncInternal(
> SequentialWriter.java:185)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.access$100(
> CompressedSequentialWriter.java:38)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter$
> TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
> ~[apache-cassandra-3.10.jar:3.10]
> at
>