Hi, The graphadmin job has to be able to connect to Cassandra in order to set up the schema, and everything else will come up once it’s complete. Can you check the health of your Cassandra nodes?
Thanks, jimmy From: <[email protected]> on behalf of "Ananthapadmanaban, Karthik via Lists.Onap.Org" <[email protected]> Reply-To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Date: Tuesday, February 5, 2019 at 8:26 AM To: "[email protected]" <[email protected]> Subject: [onap-discuss] AAI Component is not coming up using OOM - Casablanca #aai #oom #casablanca Hello All, Facing issue while bringing up AAI component recently. Is anyone facing similar issue. Kindly suggest. kubectl get pods -n onap | grep aai demo-aai-aai-7cf5dd896b-zldmb 0/1 Init:0/1 35 5h demo-aai-aai-babel-d58977746-w7k69 2/2 Running 0 5h demo-aai-aai-cassandra-0 1/1 Running 0 5h demo-aai-aai-cassandra-1 1/1 Running 1 5h demo-aai-aai-cassandra-2 1/1 Running 0 5h demo-aai-aai-champ-b4d79c8bb-j54dq 1/2 Running 0 5h demo-aai-aai-data-router-66b5547666-pjjxw 1/2 CrashLoopBackOff 72 5h demo-aai-aai-elasticsearch-569b4b7558-trfch 1/1 Running 0 5h demo-aai-aai-gizmo-84778d8f-v9hck 2/2 Running 0 5h demo-aai-aai-graphadmin-7d9cffcf6d-qnzxf 0/2 Init:0/1 35 5h demo-aai-aai-graphadmin-create-db-schema-6lbzb 1/1 Running 0 5h demo-aai-aai-graphadmin-create-db-schema-hwbsx 0/1 Error 0 5h demo-aai-aai-modelloader-67fc659c56-k7h7d 2/2 Running 0 5h demo-aai-aai-resources-994c4494c-79tmj 0/2 Init:0/1 35 5h demo-aai-aai-search-data-5cc849b8b7-2xb94 2/2 Running 0 5h demo-aai-aai-sparky-be-6c959c8569-qmps8 0/2 Init:0/1 0 5h demo-aai-aai-spike-79b755845-n2p7l 2/2 Running 0 5h demo-aai-aai-traversal-569676c469-ms8r7 0/2 Init:0/1 35 5h demo-aai-aai-traversal-update-query-data-lsgdc 0/1 Init:0/1 35 5h Below given is the log of aai-graphadmin-job Project Build Version: 1.0.1 chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/auth/aai_keystore': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/aaiconfig.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/janusgraph-realtime.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/etc/appprops/janusgraph-cached.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/localhost-access-logback.xml': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/application.properties': Read-only file system chown: changing ownership of '/opt/app/aai-graphadmin/resources/logback.xml': Read-only file system Tue Feb 5 07:23:49 UTC 2019 Starting /opt/app/aai-graphadmin/bin/createDBSchema.sh ---- NOTE --- about to open graph (takes a little while)--------; Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:595) Caused by: java.lang.ExceptionInInitializerError at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:103) at org.onap.aai.schema.GenTester.main(GenTester.java:126) ... 8 more Caused by: java.lang.RuntimeException: Failed to instantiate graphs at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:85) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:57) at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:90) ... 10 more Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.set(KCVSConfiguration.java:153) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.set(KCVSConfiguration.java:130) at org.janusgraph.diskstorage.configuration.ModifiableConfiguration.set(ModifiableConfiguration.java:40) at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:163) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113) at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:115) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:83) ... 12 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT1M at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55) ... 22 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:413) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.mutateMany(AstyanaxKeyColumnValueStore.java:280) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.mutate(AstyanaxKeyColumnValueStore.java:276) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.mutate(ExpectedValueCheckingStore.java:79) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$2.call(KCVSConfiguration.java:158) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$2.call(KCVSConfiguration.java:153) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ... 23 more Caused by: com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=10.42.236.144(10.42.236.144):9160, latency=2002(4006), attempts=2]TimedOutException(acknowledged_by:1, acknowledged_by_batchlog:true) at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28) at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153) at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:517) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:93) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:137) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:411) ... 31 more Caused by: TimedOutException(acknowledged_by:1, acknowledged_by_batchlog:true) at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29624) at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29592) at org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result.read(Cassandra.java:29526) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.cassandra.thrift.Cassandra$Client.recv_atomic_batch_mutate(Cassandra.java:1108) at org.apache.cassandra.thrift.Cassandra$Client.atomic_batch_mutate(Cassandra.java:1094) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:147) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:141) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) ... 39 more Thanks! -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15347): https://lists.onap.org/g/onap-discuss/message/15347 Mute This Topic: https://lists.onap.org/mt/29663742/21656 Mute #aai: https://lists.onap.org/mk?hashtag=aai&subid=2740164 Mute #oom: https://lists.onap.org/mk?hashtag=oom&subid=2740164 Mute #casablanca: https://lists.onap.org/mk?hashtag=casablanca&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
