Hi, Yogi, The createDbSchema job will keep trying to load the schema for JanusGraph into Cassandra until it gets a success; if AAI’s Cassandra is unavailable for any reason, you will see an error like that. Check your aai Cassandra cluster, in the oom config we specify 3 node cluster and I only see 1 in your output below, which seems strange, perhaps your environment is taking a long time to bring up Cassandra.
Thanks, jimmy From: <[email protected]> on behalf of Yogi Knss <[email protected]> Reply-To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Date: Tuesday, April 30, 2019 at 4:39 AM To: "[email protected]" <[email protected]> Subject: [onap-discuss] AAI not coming up in casablance #aai Hi, We are trying to install cassablanca on kubernetes and we are facing issues with AAI. AAi is not able to run because of issue with graphadmin, Can you let us know if we have missed some configuration ? We followed the same sequence that you have specified above. All other components "consul ,msb, dmaap, dcaegen2, aaf, robot" are up and running. The following error is seen in "aai-aai-graphadmin-create-db-schema-mjc4z" JOB. Project Build Version: 1.0.1 chown: changing ownership of '/opt/app/aai-graphadmin/resources/application.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/aaiconfig.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/janusgraph-cached.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/janusgraph-realtime.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/auth/aai_keystore': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/localhost-access-logback.xml': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/logback.xml': Read-only file system Wed Apr 24 16:09:21 IST 2019 Starting /opt/app/aai-graphadmin/bin/createDBSchema.sh ---- NOTE --- about to open graph (takes a little while)--------; Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:595) Caused by: java.lang.ExceptionInInitializerError at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:103) at org.onap.aai.schema.GenTester.main(GenTester.java:126) ... 8 more Caused by: java.lang.RuntimeException: Failed to instantiate graphs at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:85) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:57) at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:90) ... 10 more Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:100) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:106) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1394) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113) at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:115) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:82) ... 12 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT1M at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55) ... 21 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:161) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:115) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getSlice(AstyanaxKeyColumnValueStore.java:104) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:103) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ... 22 more Caused by: com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: TokenRangeOfflineException: [host=10.233.66.28(10.233.66.28):9160, latency=1(1), attempts=1]UnavailableException() at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28) at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153) at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352) at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:538) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:159) ... 29 more Caused by: UnavailableException() at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14687) at org.apache.cassandra.thrift.Cassandra$multiget_slice_result$multiget_slice_resultStandardScheme.read(Cassandra.java:14633) at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:14559) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:741) at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:725) at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:544) at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1.internalExecute(ThriftColumnFamilyQueryImpl.java:541) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) ... 35 more Failed to run the tool /opt/app/aai-graphadmin/bin/createDBSchema.sh successfully Failed to run the createDBSchema.sh Status of pods kubectl get pods -n onap NAME READY STATUS RESTARTS AGE tcs-aaf-aaf-cm-76696c8bcf-sqqpw 1/1 Running 0 7h tcs-aaf-aaf-cs-6c69c87d44-vbtk7 1/1 Running 0 7h tcs-aaf-aaf-fs-5fd8c8bd8d-fbbwb 1/1 Running 0 7h tcs-aaf-aaf-gui-777fb85d96-vtwnf 1/1 Running 0 7h tcs-aaf-aaf-hello-774f4b6f5-lb6t4 1/1 Running 0 7h tcs-aaf-aaf-locate-fbd8f454b-lgkjd 1/1 Running 0 7h tcs-aaf-aaf-oauth-77fcbf54cd-trdxx 1/1 Running 0 7h tcs-aaf-aaf-service-9f76d4746-4knwv 1/1 Running 0 7h tcs-aaf-aaf-sms-5f764cf7cf-vxl7g 1/1 Running 0 7h tcs-aaf-aaf-sms-quorumclient-0 1/1 Running 0 7h tcs-aaf-aaf-sms-quorumclient-1 1/1 Running 0 7h tcs-aaf-aaf-sms-quorumclient-2 1/1 Running 0 6h tcs-aaf-aaf-sms-vault-0 2/2 Running 3 7h tcs-aaf-aaf-sshsm-distcenter-gmbwl 0/1 Completed 0 7h tcs-aaf-aaf-sshsm-testca-69v6k 0/1 Init:Error 0 7h tcs-aaf-aaf-sshsm-testca-6ghcc 0/1 Init:Error 0 7h tcs-aaf-aaf-sshsm-testca-rz58d 0/1 Init:Error 0 7h tcs-aaf-aaf-sshsm-testca-ttbq7 0/1 Init:Error 0 7h tcs-aaf-aaf-sshsm-testca-xht79 0/1 Completed 0 6h tcs-aaf-aaf-sshsm-testca-z6sfw 0/1 Init:Error 0 7h tcs-aai-aai-5cbbdb4ff5-4hwpb 0/1 Init:0/1 45 7h tcs-aai-aai-babel-68fc787d74-pvdtr 2/2 Running 0 1d tcs-aai-aai-cassandra-0 1/1 Running 1 1d tcs-aai-aai-champ-5b64bd67c7-qnx4j 1/2 Running 0 1d tcs-aai-aai-data-router-6654597bdd-6nfmw 2/2 Running 6 1d tcs-aai-aai-elasticsearch-68bcf6c8fd-9stb4 1/1 Running 0 1d tcs-aai-aai-gizmo-d548b4d9-cvf6d 2/2 Running 0 1d tcs-aai-aai-graphadmin-67f5965fb7-88dwz 0/2 Init:0/1 165 1d tcs-aai-aai-graphadmin-create-db-schema-mjc4z 0/1 Error 0 1d tcs-aai-aai-graphadmin-create-db-schema-p5vms 0/1 Error 0 1d tcs-aai-aai-graphadmin-create-db-schema-sbhsk 0/1 Error 0 1d tcs-aai-aai-graphadmin-create-db-schema-wqw29 0/1 Error 0 1d tcs-aai-aai-graphadmin-create-db-schema-zrmpv 0/1 Error 0 1d tcs-aai-aai-modelloader-6dfbcd7596-9dd9s 2/2 Running 0 1d tcs-aai-aai-resources-588998b4ff-qnlvd 0/2 Init:0/1 165 1d tcs-aai-aai-search-data-57666c9494-n24bc 2/2 Running 0 1d tcs-aai-aai-sparky-be-7db4b8dcf-qr29r 0/2 Init:0/1 0 1d tcs-aai-aai-spike-6f9f5f5c9d-fxcrs 2/2 Running 0 1d tcs-aai-aai-traversal-7df69d5885-72kj5 0/2 Init:0/1 165 1d tcs-aai-aai-traversal-update-query-data-gr2w8 0/1 Init:0/1 165 1d tcs-consul-consul-69d7c64bdd-wms5l 1/1 Running 0 1d tcs-consul-consul-server-0 1/1 Running 0 1d tcs-consul-consul-server-1 1/1 Running 0 1d tcs-consul-consul-server-2 1/1 Running 0 1d tcs-dcaegen2-dcae-bootstrap-6b6bb89cd5-5vhmv 1/1 Running 0 1d tcs-dcaegen2-dcae-cloudify-manager-6b6f59fc66-k79c9 1/1 Running 0 1d tcs-dcaegen2-dcae-db-0 1/1 Running 0 1d tcs-dcaegen2-dcae-db-1 1/1 Running 0 1d tcs-dcaegen2-dcae-healthcheck-5fc6d94989-ch5kn 1/1 Running 0 1d tcs-dcaegen2-dcae-pgpool-77b844664d-5hd4c 1/1 Running 0 1d tcs-dcaegen2-dcae-pgpool-77b844664d-xxnh8 1/1 Running 0 1d tcs-dcaegen2-dcae-redis-0 1/1 Running 0 1d tcs-dcaegen2-dcae-redis-1 1/1 Running 0 1d tcs-dcaegen2-dcae-redis-2 1/1 Running 0 1d tcs-dcaegen2-dcae-redis-3 1/1 Running 0 1d tcs-dcaegen2-dcae-redis-4 1/1 Running 0 1d tcs-dcaegen2-dcae-redis-5 1/1 Running 0 1d tcs-dmaap-dbc-pg-0 1/1 Running 0 1d tcs-dmaap-dbc-pg-1 1/1 Running 0 1d tcs-dmaap-dbc-pgpool-57d7b76446-qgrcs 1/1 Running 0 1d tcs-dmaap-dbc-pgpool-57d7b76446-qphtf 1/1 Running 0 1d tcs-dmaap-dmaap-bus-controller-7567b865b7-x7vtz 1/1 Running 0 1d tcs-dmaap-dmaap-dr-db-655587488d-2j5b2 1/1 Running 1 1d tcs-dmaap-dmaap-dr-node-649659c584-ctfjs 1/1 Running 0 1d tcs-dmaap-dmaap-dr-prov-595cd8bc55-6kjtv 1/1 Running 6 1d tcs-dmaap-message-router-5f7b985c88-dj8j2 1/1 Running 0 1d tcs-dmaap-message-router-kafka-678fb8558b-hc768 1/1 Running 0 1d tcs-dmaap-message-router-zookeeper-54bb8cd9cf-kl55w 1/1 Running 0 1d tcs-esr-esr-gui-699d9f579b-hrhk9 1/1 Running 0 7h tcs-esr-esr-server-85d8c5f57-vj7m4 2/2 Running 0 7h tcs-msb-kube2msb-6db5fd8c85-xrbs7 1/1 Running 0 1d tcs-msb-msb-consul-66445944b6-phjpk 1/1 Running 0 1d tcs-msb-msb-discovery-6bd858b659-42xbp 2/2 Running 0 1d tcs-msb-msb-eag-78fbb94cc9-7cn2p 2/2 Running 0 1d tcs-msb-msb-iag-5d45c9999b-lwcb6 2/2 Running 0 1d tcs-oof-cmso-db-0 0/1 CrashLoopBackOff 105 7h tcs-oof-music-cassandra-0 1/1 Running 0 7h tcs-oof-music-cassandra-1 1/1 Running 0 7h tcs-oof-music-cassandra-2 1/1 Running 0 7h tcs-oof-music-cassandra-job-config-z28dh 0/1 Completed 0 7h tcs-oof-music-tomcat-8f64f65d8-4pbnp 1/1 Running 0 7h tcs-oof-music-tomcat-8f64f65d8-dl58x 1/1 Running 0 7h tcs-oof-music-tomcat-8f64f65d8-fmrjn 0/1 Init:0/3 0 7h tcs-oof-oof-557c8c677d-rjb2b 0/1 Init:0/2 45 7h tcs-oof-oof-cmso-service-5467475444-ppmd7 0/1 Init:0/2 45 7h tcs-oof-oof-has-api-5d79d97fb7-nzfkx 0/1 Init:0/3 45 7h tcs-oof-oof-has-controller-658bbb894-m9dqf 0/1 Init:0/3 45 7h tcs-oof-oof-has-data-5575788564-zbv4x 0/1 Init:0/4 45 7h tcs-oof-oof-has-healthcheck-k5swr 0/1 Init:0/1 45 7h tcs-oof-oof-has-onboard-bvfsz 0/1 Init:0/2 45 7h tcs-oof-oof-has-reservation-9fd696d8d-dbw2k 0/1 Init:0/4 45 7h tcs-oof-oof-has-solver-7fd7878df9-dn4xw 0/1 Init:0/4 45 7h tcs-oof-zookeeper-0 1/1 Running 0 7h tcs-oof-zookeeper-1 1/1 Running 0 7h tcs-oof-zookeeper-2 1/1 Running 0 7h tcs-robot-robot-86d89ffdb9-xd65b 1/1 Running 0 1d Regards, Yogi -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16820): https://lists.onap.org/g/onap-discuss/message/16820 Mute This Topic: https://lists.onap.org/mt/31418159/21656 Mute #aai: https://lists.onap.org/mk?hashtag=aai&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
