Dear All, So module's some pods are crashing like below, I tried to recreate the pods several times however issue is persisting.
dev-so-so-5d9c4fbf5c-lwq9l 0/1 CrashLoopBackOff 21 1h dev-so-so-bpmn-infra-5675c9fd55-jbpxg 0/1 CrashLoopBackOff 19 1h dev-so-so-catalog-db-adapter-95f9cc64-mtxr2 0/1 CrashLoopBackOff 20 1h dev-so-so-mariadb-5f854cbbbb-jwnvt 1/1 Running 0 1h dev-so-so-monitoring-6c76f45c4f-m9mcn 1/1 Running 0 1h dev-so-so-openstack-adapter-64cc5554fb-cdlll 0/1 CrashLoopBackOff 21 1h dev-so-so-request-db-adapter-87b79c98d-kx6mf 0/1 CrashLoopBackOff 20 1h dev-so-so-sdc-controller-6ddd49b8b-6vn7f 0/1 CrashLoopBackOff 20 1h dev-so-so-sdnc-adapter-747bfc998f-s99r7 1/1 Running 0 1h dev-so-so-vfc-adapter-6b4cf8c449-5s7g2 0/1 CrashLoopBackOff 21 1h All those crashing pods are dependent to so-mariadb container. It is running and here are the logs form mariadb: ubuntu@kub4:~$ kubectl logs -f dev-so-so-mariadb-5f854cbbbb-jwnvt -n onap 2018-12-24 10:58:32 140282828376000 [Note] mysqld (mysqld 10.1.11-MariaDB-1~jessie-log) starting as process 1 ... 2018-12-24 10:58:33 140282828376000 [Note] WSREP: Read nil XID from storage engines, skipping position init 2018-12-24 10:58:33 140282828376000 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2018-12-24 10:58:33 140282828376000 [Note] WSREP: wsrep_load(): Galera 25.3.12(r3516) by Codership Oy <[email protected]> loaded successfully. 2018-12-24 10:58:33 140282828376000 [Note] WSREP: CRC-32C: using hardware acceleration. 2018-12-24 10:58:33 140282828376000 [Note] WSREP: Found saved state: 4c21e5c7-04f9-11e9-939c-53aee0b02696:0 2018-12-24 10:58:33 140282828376000 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.42.171.249; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 1G; gcache.size = 2G; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc 2018-12-24 10:58:33 140280556070656 [Note] WSREP: Service thread queue flushed. 2018-12-24 10:58:33 140282828376000 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1 2018-12-24 10:58:33 140282828376000 [Note] WSREP: wsrep_sst_grab() 2018-12-24 10:58:33 140282828376000 [Note] WSREP: Start replication 2018-12-24 10:58:33 140282828376000 [Note] WSREP: Setting initial position to 4c21e5c7-04f9-11e9-939c-53aee0b02696:0 2018-12-24 10:58:33 140282828376000 [Note] WSREP: protonet asio version 0 2018-12-24 10:58:33 140282828376000 [Note] WSREP: Using CRC-32C for message checksums. 2018-12-24 10:58:33 140282828376000 [Note] WSREP: backend: asio 2018-12-24 10:58:34 140282828376000 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2018-12-24 10:58:34 140282828376000 [Note] WSREP: restore pc from disk failed 2018-12-24 10:58:34 140282828376000 [Note] WSREP: GMCast version 0 2018-12-24 10:58:34 140282828376000 [Note] WSREP: (db6894c8, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2018-12-24 10:58:34 140282828376000 [Note] WSREP: (db6894c8, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2018-12-24 10:58:34 140282828376000 [Note] WSREP: EVS version 0 2018-12-24 10:58:34 140282828376000 [Note] WSREP: gcomm: connecting to group 'MSO-automated-tests-cluster', peer '' 2018-12-24 10:58:34 140282828376000 [Note] WSREP: start_prim is enabled, turn off pc_recovery 2018-12-24 10:58:34 140282828376000 [Note] WSREP: Node db6894c8 state prim 2018-12-24 10:58:34 140282828376000 [Note] WSREP: view(view_id(PRIM,db6894c8,1) memb { db6894c8,0 } joined { } left { } partitioned { }) 2018-12-24 10:58:34 140282828376000 [Note] WSREP: save pc into disk 2018-12-24 10:58:34 140282828376000 [Note] WSREP: gcomm: connected 2018-12-24 10:58:34 140282828376000 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2018-12-24 10:58:34 140282828376000 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2018-12-24 10:58:34 140282828376000 [Note] WSREP: Opened channel 'MSO-automated-tests-cluster' 2018-12-24 10:58:34 140280485377792 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 2018-12-24 10:58:34 140280485377792 [Note] WSREP: STATE_EXCHANGE: sent state UUID: db79d47b-076a-11e9-8766-a204113915a7 2018-12-24 10:58:34 140280485377792 [Note] WSREP: STATE EXCHANGE: sent state msg: db79d47b-076a-11e9-8766-a204113915a7 2018-12-24 10:58:34 140280485377792 [Note] WSREP: STATE EXCHANGE: got state msg: db79d47b-076a-11e9-8766-a204113915a7 from 0 (mariadb1) 2018-12-24 10:58:34 140280485377792 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf_id = 0, members = 1/1 (joined/total), act_id = 0, last_appl. = -1, protocols = 0/7/3 (gcs/repl/appl), group UUID = 4c21e5c7-04f9-11e9-939c-53aee0b02696 2018-12-24 10:58:34 140280485377792 [Note] WSREP: Flow-control interval: [16, 16] 2018-12-24 10:58:34 140280485377792 [Note] WSREP: Restored state OPEN -> JOINED (0) 2018-12-24 10:58:34 140280485377792 [Note] WSREP: Member 0.0 (mariadb1) synced with group. 2018-12-24 10:58:34 140280485377792 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0) 2018-12-24 10:58:34 140282828376000 [Note] WSREP: Waiting for SST to complete. 2018-12-24 10:58:34 140282827052800 [Note] WSREP: New cluster view: global state: 4c21e5c7-04f9-11e9-939c-53aee0b02696:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3 2018-12-24 10:58:34 140282828376000 [Note] WSREP: SST complete, seqno: 0 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: The InnoDB memory heap is disabled 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Memory barrier is not used 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Compressed tables use zlib 1.2.8 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Using Linux native AIO 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Using SSE crc32 instructions 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Initializing buffer pool, size = 256.0M 2018-12-24 10:58:34 140282828376000 [Note] InnoDB: Completed initialization of buffer pool 2018-12-24 10:58:35 140282828376000 [Note] InnoDB: Highest supported file format is Barracuda. 2018-12-24 10:58:41 140282828376000 [Note] InnoDB: 128 rollback segment(s) are active. 2018-12-24 10:58:41 140282828376000 [Note] InnoDB: Waiting for purge to start 2018-12-24 10:58:41 140282828376000 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-76.0 started; log sequence number 7024327 2018-12-24 10:58:41 140279965255424 [Note] InnoDB: Dumping buffer pool(s) not yet started 2018-12-24 10:58:41 140282828376000 [Note] Plugin 'FEEDBACK' is disabled. 181224 10:58:41 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED. 181224 10:58:41 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled.181224 10:58:41 server_audit: logging started to the file //var/lib/mysql/audit.log. 2018-12-24 10:58:42 140282828376000 [Note] Server socket created on IP: '::'. 2018-12-24 10:58:42 140282828376000 [Warning] 'proxies_priv' entry '@% root@mariadb' ignored in --skip-name-resolve mode. 2018-12-24 10:58:44 140282827052800 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2018-12-24 10:58:44 140282827052800 [Note] WSREP: REPL Protocols: 7 (3, 2) 2018-12-24 10:58:44 140280556070656 [Note] WSREP: Service thread queue flushed. 2018-12-24 10:58:44 140282827052800 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3 2018-12-24 10:58:44 140280556070656 [Note] WSREP: Service thread queue flushed. 2018-12-24 10:58:44 140282827052800 [Note] WSREP: Synchronized with group, ready for connections 2018-12-24 10:58:44 140282827052800 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2018-12-24 10:58:44 140282828376000 [Note] mysqld: ready for connections. Version: '10.1.11-MariaDB-1~jessie-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution Crashing container logs: ubuntu@kub4:~$ kubectl logs -f dev-so-so-request-db-adapter-87b79c98d-kx6mf -n onap Installing onap-ca.crt in /usr/local/share/ca-certificates WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping JVM Arguments: -Djava.security.egd=file:/dev/./urandom -Dlogs_dir=./logs/reqdb/ -Dlogging.config=/app/logback-spring.xml -Dspring.config.location=/app/config/override.yaml . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v1.5.13.RELEASE) Application exiting with status code 1 ************************************************************* ubuntu@kub3:~$ kubectl logs -f dev-so-so-bpmn-infra-5675c9fd55-jbpxg -n onap Installing onap-ca.crt in /usr/local/share/ca-certificates WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping JVM Arguments: -Djava.security.egd=file:/dev/./urandom -Dlogs_dir=./logs/bpmn/ -Dlogging.config=/app/logback-spring.xml -Dspring.config.location=/app/config/override.yaml ____ _ ____ ____ __ __ / ___| __ _ _ __ ___ _ _ _ __ __| | __ _ | __ )| _ \| \/ | | | / _` | '_ ` _ \| | | | '_ \ / _` |/ _` | | _ \| |_) | |\/| | | |__| (_| | | | | | | |_| | | | | (_| | (_| | | |_) | __/| | | | \____/\__,_|_| |_| |_|\__,_|_| |_|\__,_|\__,_| |____/|_| |_| |_| Spring-Boot: (v1.5.13.RELEASE) Camunda BPM: (v7.8.0) Camunda BPM Spring Boot Starter: (v2.3.0) Application exiting with status code 1 ************************************************* ubuntu@kub3:~$ kubectl logs -f dev-so-so-catalog-db-adapter-95f9cc64-mtxr2 -n onap Installing onap-ca.crt in /usr/local/share/ca-certificates WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping JVM Arguments: -Djava.security.egd=file:/dev/./urandom -Dlogs_dir=./logs/catdb/ -Dlogging.config=/app/logback-spring.xml -Dspring.config.location=/app/config/override.yaml . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v1.5.13.RELEASE) Application exiting with status code 1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14686): https://lists.onap.org/g/onap-discuss/message/14686 Mute This Topic: https://lists.onap.org/mt/28717146/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
