[jira] [Updated] (CASSANDRA-10964) Startup errors in Docker containers depending on memtable allocation type
[
https://issues.apache.org/jira/browse/CASSANDRA-10964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy Hanna updated CASSANDRA-10964:
-
Component/s: (was: Materialized Views)
> Startup errors in Docker containers depending on memtable allocation type
> -
>
> Key: CASSANDRA-10964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10964
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
> Environment: Docker, Debian Testing, 3.0.1
>Reporter: Jacek Furmankiewicz
>
> We are creating Docker containers for various versions of Cassandra. All are
> based on Debian, Oracle JDK 1.8 and the Cassandra versions are installed
> directly from the DataStax Debian repos via apt-get.
> We noticed that with 3.0.1 (only that version, 2.1.11 and 2.2.4 work always
> fine) the Cassandra process fails to start up randonly (about 50% of the
> time) with the following error:
> {noformat}
> Caused by: java.lang.RuntimeException:
> system_distributed:parent_repair_history not found in the schema definitions
> keyspace.
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:940)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:931)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:894)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:886)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1276)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1255)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> {noformat}
> Started playing with different configuration parameters and by trial and
> error figured out it seems to be related to this configuration parameter:
> {noformat}
> memtable_allocation_type: offheap_buffers
> {noformat}
> If we set it to offheap_buffers, this error occurs about 50% of the time
> (when starting on a new clean filesystem).
> If we set it to heap_buffers, it works always, 100% of the time, never an
> issue.
> Attaching full stack output to help debug:
> {noformat}
> INFO 16:11:44 Configuration location:
> file:/etc/cassandra/cassandra.yaml
> INFO 16:11:44 Node
> configuration:[authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_snapshot=true;
> batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5;
> batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000;
> client_encryption_options=; cluster_name=TEST_CLUSTER;
> column_index_size_in_kb=64; commit_failure_policy=stop;
> commitlog_directory=/var/lib/cassandra/commitlog2;
> commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> commitlog_sync_period_in_ms=1;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_counter_writes=12;
> concurrent_materialized_view_writes=7; concurrent_reads=64;
> concurrent_writes=10; counter_cache_save_period=17200;
> counter_cache_size_in_mb=1027; counter_write_request_timeout_in_ms=5000;
> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data2];
> disk_failure_policy=stop; disk_optimization_strategy=spinning;
> dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100;
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false; endpoint_snitch=SimpleSnitch;
> gc_warn_threshold_in_ms=1000; hinted_handoff_enabled=true;
> hinted_handoff_throttle_in_kb=1024;
> hints_directory=/var/lib/cassandra/hints2; hints_flush_period_in_ms=1;
>
[jira] [Updated] (CASSANDRA-10964) Startup errors in Docker containers depending on memtable allocation type
[
https://issues.apache.org/jira/browse/CASSANDRA-10964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy Hanna updated CASSANDRA-10964:
-
Component/s: Materialized Views
> Startup errors in Docker containers depending on memtable allocation type
> -
>
> Key: CASSANDRA-10964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10964
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration, Materialized Views
> Environment: Docker, Debian Testing, 3.0.1
>Reporter: Jacek Furmankiewicz
>
> We are creating Docker containers for various versions of Cassandra. All are
> based on Debian, Oracle JDK 1.8 and the Cassandra versions are installed
> directly from the DataStax Debian repos via apt-get.
> We noticed that with 3.0.1 (only that version, 2.1.11 and 2.2.4 work always
> fine) the Cassandra process fails to start up randonly (about 50% of the
> time) with the following error:
> {noformat}
> Caused by: java.lang.RuntimeException:
> system_distributed:parent_repair_history not found in the schema definitions
> keyspace.
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:940)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:931)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:894)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:886)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1276)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1255)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> {noformat}
> Started playing with different configuration parameters and by trial and
> error figured out it seems to be related to this configuration parameter:
> {noformat}
> memtable_allocation_type: offheap_buffers
> {noformat}
> If we set it to offheap_buffers, this error occurs about 50% of the time
> (when starting on a new clean filesystem).
> If we set it to heap_buffers, it works always, 100% of the time, never an
> issue.
> Attaching full stack output to help debug:
> {noformat}
> INFO 16:11:44 Configuration location:
> file:/etc/cassandra/cassandra.yaml
> INFO 16:11:44 Node
> configuration:[authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_snapshot=true;
> batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5;
> batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000;
> client_encryption_options=; cluster_name=TEST_CLUSTER;
> column_index_size_in_kb=64; commit_failure_policy=stop;
> commitlog_directory=/var/lib/cassandra/commitlog2;
> commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> commitlog_sync_period_in_ms=1;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_counter_writes=12;
> concurrent_materialized_view_writes=7; concurrent_reads=64;
> concurrent_writes=10; counter_cache_save_period=17200;
> counter_cache_size_in_mb=1027; counter_write_request_timeout_in_ms=5000;
> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data2];
> disk_failure_policy=stop; disk_optimization_strategy=spinning;
> dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100;
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false; endpoint_snitch=SimpleSnitch;
> gc_warn_threshold_in_ms=1000; hinted_handoff_enabled=true;
> hinted_handoff_throttle_in_kb=1024;
> hints_directory=/var/lib/cassandra/hints2;
[jira] [Updated] (CASSANDRA-10964) Startup errors in Docker containers depending on memtable allocation type
[
https://issues.apache.org/jira/browse/CASSANDRA-10964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacek Furmankiewicz updated CASSANDRA-10964:
Summary: Startup errors in Docker containers depending on memtable
allocation type (was: Starup errors in Docker containers depending on memtable
allocation type)
> Startup errors in Docker containers depending on memtable allocation type
> -
>
> Key: CASSANDRA-10964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10964
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
> Environment: Docker, Debian Testing, 3.0.1
>Reporter: Jacek Furmankiewicz
>
> We are creating Docker containers for various versions of Cassandra. All are
> based on Debian, Oracle JDK 1.8 and the Cassandra versions are installed
> directly from the DataStax Debian repos via apt-get.
> We noticed that with 3.0.1 (only that version, 2.1.11 and 2.2.4 work always
> fine) the Cassandra process fails to start up randonly (about 50% of the
> time) with the following error:
> {noformat}
> Caused by: java.lang.RuntimeException:
> system_distributed:parent_repair_history not found in the schema definitions
> keyspace.
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:940)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:931)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:894)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesOnly(SchemaKeyspace.java:886)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1276)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1255)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:531)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.0.1.jar:3.0.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> {noformat}
> Started playing with different configuration parameters and by trial and
> error figured out it seems to be related to this configuration parameter:
> {noformat}
> memtable_allocation_type: offheap_buffers
> {noformat}
> If we set it to offheap_buffers, this error occurs about 50% of the time
> (when starting on a new clean filesystem).
> If we set it to heap_buffers, it works always, 100% of the time, never an
> issue.
> Attaching full stack output to help debug:
> {noformat}
> INFO 16:11:44 Configuration location:
> file:/etc/cassandra/cassandra.yaml
> INFO 16:11:44 Node
> configuration:[authenticator=PasswordAuthenticator;
> authorizer=CassandraAuthorizer; auto_snapshot=true;
> batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5;
> batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000;
> client_encryption_options=; cluster_name=TEST_CLUSTER;
> column_index_size_in_kb=64; commit_failure_policy=stop;
> commitlog_directory=/var/lib/cassandra/commitlog2;
> commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
> commitlog_sync_period_in_ms=1;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=16; concurrent_counter_writes=12;
> concurrent_materialized_view_writes=7; concurrent_reads=64;
> concurrent_writes=10; counter_cache_save_period=17200;
> counter_cache_size_in_mb=1027; counter_write_request_timeout_in_ms=5000;
> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data2];
> disk_failure_policy=stop; disk_optimization_strategy=spinning;
> dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100;
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false; endpoint_snitch=SimpleSnitch;
> gc_warn_threshold_in_ms=1000;
