Hello All, Facing issue while bringing up AAI component recently. Is anyone facing similar issue.
Kindly suggest. *kubectl get pods -n onap | grep aai* demo-aai-aai-7cf5dd896b-zldmb 0/1 Init:0/1 35 5h demo-aai-aai-babel-d58977746-w7k69 2/2 Running 0 5h demo-aai-aai-cassandra-0 1/1 Running 0 5h demo-aai-aai-cassandra-1 1/1 Running 1 5h demo-aai-aai-cassandra-2 1/1 Running 0 5h demo-aai-aai-champ-b4d79c8bb-j54dq 1/2 Running 0 5h demo-aai-aai-data-router-66b5547666-pjjxw 1/2 CrashLoopBackOff 72 5h demo-aai-aai-elasticsearch-569b4b7558-trfch 1/1 Running 0 5h demo-aai-aai-gizmo-84778d8f-v9hck 2/2 Running 0 5h demo-aai-aai-graphadmin-7d9cffcf6d-qnzxf 0/2 Init:0/1 35 5h demo-aai-aai-graphadmin-create-db-schema-6lbzb 1/1 Running 0 5h demo-aai-aai-graphadmin-create-db-schema-hwbsx 0/1 Error 0 5h demo-aai-aai-modelloader-67fc659c56-k7h7d 2/2 Running 0 5h demo-aai-aai-resources-994c4494c-79tmj 0/2 Init:0/1 35 5h demo-aai-aai-search-data-5cc849b8b7-2xb94 2/2 Running 0 5h demo-aai-aai-sparky-be-6c959c8569-qmps8 0/2 Init:0/1 0 5h demo-aai-aai-spike-79b755845-n2p7l 2/2 Running 0 5h demo-aai-aai-traversal-569676c469-ms8r7 0/2 Init:0/1 35 5h demo-aai-aai-traversal-update-query-data-lsgdc 0/1 Init:0/1 35 5h *Below given is the log of aai-graphadmin-job* Project Build Version: 1.0.1 chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/auth/aai_keystore': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/aaiconfig.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/janusgraph-realtime.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/janusgraph-cached.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/localhost-access-logback.xml': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/application.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/logback.xml': Read-only file system Tue Feb 5 07:23:49 UTC 2019 Starting /opt/app/aai-graphadmin/bin/createDBSchema.sh ---- NOTE --- about to open graph (takes a little while)--------; Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:595) Caused by: java.lang.ExceptionInInitializerError at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:103) at org.onap.aai.schema.GenTester.main(GenTester.java:126) ... 8 more Caused by: java.lang.RuntimeException: Failed to instantiate graphs at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:85) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:57) at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:90) ... 10 more Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.set(KCVSConfiguration.java:153) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.set(KCVSConfiguration.java:130) at org.janusgraph.diskstorage.configuration.ModifiableConfiguration.set(ModifiableConfiguration.java:40) at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:163) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113) at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:115) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:83) ... 12 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT1M at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55) ... 22 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:413) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.mutateMany(AstyanaxKeyColumnValueStore.java:280) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.mutate(AstyanaxKeyColumnValueStore.java:276) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.mutate(ExpectedValueCheckingStore.java:79) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$2.call(KCVSConfiguration.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$2.call(KCVSConfiguration.java:153) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ... 23 more Caused by: com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=10.42.236.144(10.42.236.144):9160, latency=2002(4006), attempts=2]TimedOutException(acknowledged_by:1, acknowledged_by_batchlog:true) at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28) at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153) at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:517) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:93) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:137) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:411) ... 31 more Caused by: TimedOutException(acknowledged_by:1, acknowledged_by_batchlog:true) at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29624) at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29592) at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result.read(Cassandra.java:29526) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.cassandra.thrift.Cassandra$Client.recv_atomic_batch_mutate(Cassandra.java:1108) at org.apache.cassandra.thrift.Cassandra$Client.atomic_batch_mutate(Cassandra.java:1094) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:147) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:141) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) ... 39 more Thanks! -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15338): https://lists.onap.org/g/onap-discuss/message/15338 Mute This Topic: https://lists.onap.org/mt/29663742/21656 Mute #aai: https://lists.onap.org/mk?hashtag=aai&subid=2740164 Mute #oom: https://lists.onap.org/mk?hashtag=oom&subid=2740164 Mute #casablanca: https://lists.onap.org/mk?hashtag=casablanca&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
