Hi guys,
I'm wondering if someone could point me out in right direction. I deployed
OpenShift from Ansible playbook. All seems to be working the way I expect
of the metrics. I deleted previous deployment which didn't work and tried
different variations using:
ansible-playbook
openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml -e
openshift_metrics_install_metrics=True -e
openshift_metrics_hawkular_hostname=hawkular-metrics.os.hr4.local -e
openshift_metrics_cassandra_storage_type=pv -e
openshift_metrics_duration=14 -e openshift_metrics_start_cluster -e
openshift_metrics_image_version=v1.5.1
so when I put the version I tried with the latest, I tried with 1.4.1,
1.3.1 and 1.5.1 same result.
[root@master ~]# oc get pods
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-z2f77 0/1 CrashLoopBackOff 12 53m
hawkular-metrics-j8zgn 0/1 CrashLoopBackOff 24 53m
heapster-1qc5g 0/1 Running 6 53m
Cassandra logs
[root@master ~]# oc logs hawkular-cassandra-1-z2f77
The MAX_HEAP_SIZE envar is not set. Basing the MAX_HEAP_SIZE on the
available memory limit for the pod (2000000000).
The memory limit is less than 2GB. Using 1/2 of available memory for the
max_heap_size.
The MAX_HEAP_SIZE has been set to 953M
THE HEAP_NEWSIZE envar is not set. Setting to 1600M based on the CPU_LIMIT
of 16000. [100M per CPU core]
About to generate seeds
Trying to access the Seed list [try #1]
Setting seeds to be ;; connection timed out; trying next origin,;;
connection timed out; trying next origin,;; connection timed out; trying
next origin,;; connection timed out; trying next origin
sed: -e expression #1, char 13: unterminated `s' command
cat: /etc/ld.so.conf.d/*.conf: No such file or directory
getopt: invalid option -- 'R'
OpenJDK 64-Bit Server VM warning: Cannot open file
/opt/apache-cassandra/logs/gc.log due to No such file or directory
OpenJDK 64-Bit Server VM warning: MaxNewSize (1638400k) is equal to or
greater than the entire heap (976896k). A new max generation size of
976832k will be used.
CompilerOracle: dontinline
org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset
(Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
CompilerOracle: dontinline
org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset
(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: dontinline
org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize
(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
CompilerOracle: dontinline
org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
CompilerOracle: dontinline
org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
CompilerOracle: dontinline
org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
CompilerOracle: dontinline
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
CompilerOracle: dontinline
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow
()V
CompilerOracle: dontinline
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
CompilerOracle: dontinline
org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds
(JJ)V
CompilerOracle: inline
org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary
(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline
org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan
(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
(Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
INFO [main] 2017-07-28 15:03:28,947 YamlConfigurationLoader.java:85 -
Configuration location:
file:/opt/apache-cassandra-3.0.9-1/conf/cassandra.yaml
INFO [main] 2017-07-28 15:03:29,098 Config.java:446 - Node
configuration:[allocate_tokens_for_keyspace=null;
authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer;
auto_bootstrap=true; auto_snapshot=true;
batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5;
batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true;
cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
cluster_name=hawkular-metrics; column_index_size_in_kb=64;
commit_failure_policy=stop; commitlog_compression=LZ4Compressor;
commitlog_directory=/cassandra_data/commitlog;
commitlog_max_compression_buffers_in_pool=3;
commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null;
commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null;
compaction_large_partition_warning_threshold_mb=100;
compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32;
counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@482cd91f;
disk_access_mode=auto; disk_failure_policy=stop;
disk_optimization_estimate_percentile=0.95;
disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
dynamic_snitch_reset_interval_in_ms=600000;
dynamic_snitch_update_interval_in_ms=100;
enable_scripted_user_defined_functions=false;
enable_user_defined_functions=false;
enable_user_defined_functions_threads=true; encryption_options=null;
endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=512;
gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000;
hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true;
hinted_handoff_throttle_in_kb=1024; hints_compression=null;
hints_directory=null; hints_flush_period_in_ms=10000;
incremental_backups=false; index_interval=null;
index_summary_capacity_in_mb=null;
index_summary_resize_interval_in_minutes=60; initial_token=null;
inter_dc_stream_throughput_outbound_megabits_per_sec=200;
inter_dc_tcp_nodelay=false; internode_authenticator=null;
internode_compression=all; internode_recv_buff_size_in_bytes=null;
internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647;
key_cache_save_period=14400; key_cache_size_in_mb=null;
listen_address=hawkular-cassandra-1-z2f77; listen_interface=null;
listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
max_streaming_retries=3; max_value_size_in_mb=256;
memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null;
memtable_flush_writers=null; memtable_heap_space_in_mb=null;
memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50;
native_transport_max_concurrent_connections=-1;
native_transport_max_concurrent_connections_per_ip=-1;
native_transport_max_frame_size_in_mb=256;
native_transport_max_threads=128; native_transport_port=9042;
native_transport_port_ssl=null; num_tokens=256;
otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200;
partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1;
permissions_validity_in_ms=2000; phi_convict_threshold=8.0;
range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000;
request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
request_scheduler_id=null; request_scheduler_options=null;
request_timeout_in_ms=10000; role_manager=CassandraRoleManager;
roles_cache_max_entries=1000; roles_update_interval_in_ms=-1;
roles_validity_in_ms=2000;
row_cache_class_name=org.apache.cassandra.cache.OHCProvider;
row_cache_keys_to_save=2147483647; row_cache_save_period=0;
row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-z2f77;
rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true;
rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16;
rpc_port=9160; rpc_recv_buff_size_in_bytes=null;
rpc_send_buff_size_in_bytes=null; rpc_server_type=sync;
saved_caches_directory=null;
seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=${SEEDS}};
server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
start_native_transport=true; start_rpc=false; storage_port=7000;
stream_throughput_outbound_megabits_per_sec=200;
streaming_socket_timeout_in_ms=86400000;
thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16;
tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
tracetype_query_ttl=86400; tracetype_repair_ttl=604800;
trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
truncate_request_timeout_in_ms=60000;
unlogged_batch_across_partitions_warn_threshold=10;
user_defined_function_fail_timeout=1500;
user_defined_function_warn_timeout=500; user_function_timeout_policy=die;
windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO [main] 2017-07-28 15:03:29,099 DatabaseDescriptor.java:311 -
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2017-07-28 15:03:29,256 DatabaseDescriptor.java:418 - Global
memtable on-heap threshold is enabled at 214MB
INFO [main] 2017-07-28 15:03:29,256 DatabaseDescriptor.java:422 - Global
memtable off-heap threshold is enabled at 214MB
WARN [main] 2017-07-28 15:03:29,278 DatabaseDescriptor.java:543 - Small
commitlog volume detected at /cassandra_data/commitlog; setting
commitlog_total_space_in_mb to 4250. You can override this in
cassandra.yaml
WARN [main] 2017-07-28 15:03:29,280 DatabaseDescriptor.java:600 - Only
14457 MB free across all data volumes. Consider adding more capacity to
your cluster or removing obsolete snapshots
In regards to hawkular metrics pod:
[root@master ~]# oc logs -f hawkular-metrics-j8zgn
2017-07-28 15:04:57 Starting Hawkular Metrics
[root@master ~]#
all the configuration seems to be OK. I have no idea where to go next
really.
Thank you.
Regards,
TH
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
