David/ Keong,
Thanks for your response.
In the process of bringing the set up back up, I am facing exception in
aai-aai-graphadmin-create-db-schema job and aai pods would not start.
I did a fresh helm deploy (sudo helm deploy demo ./onap --namespace onap) using
new rancher. Cleared the /dockerdata-nfs. So it doesn’t have any stale entries.
I am using Casablanca oom kubernetes deployment environment. The same thing was
working fine earlier and I haven’t changed anything.
Please find the exception attached. It fails to instantiate Graph. I also see
same exception when I try to execute the command sh createDBSchema.sh inside
the container with aaiadmin user.
Please find the aai pods below.
aaiadmin@demo-aai-aai-graphadmin-create-db-schema-mh4l9:/opt/app/aai-graphadmin/bin$
sh createDBSchema.sh
demo-aai-aai-75b9c64998-tdkrc 0/1 Init:0/1
2 25m
demo-aai-aai-babel-5499b48894-f8qf2 2/2 Running
0 25m
demo-aai-aai-cassandra-0 1/1 Running
0 25m
demo-aai-aai-cassandra-1 1/1 Running
1 22m
demo-aai-aai-cassandra-2 1/1 Running
0 21m
demo-aai-aai-champ-75f5d6f664-26295 1/2 Running
0 25m
demo-aai-aai-data-router-94f79b78f-dqp8l 2/2 Running
0 25m
demo-aai-aai-elasticsearch-f868cff98-p4grb 1/1 Running
0 25m
demo-aai-aai-gizmo-7b6ff8f6f-lkxwf 2/2 Running
0 25m
demo-aai-aai-graphadmin-8485cd6c85-bls7f 0/2 Init:0/1
2 25m
demo-aai-aai-graphadmin-create-db-schema-mh4l9 1/1 Running
0 51s
demo-aai-aai-modelloader-77496bd596-wr9tp 2/2 Running
0 25m
demo-aai-aai-resources-6d8d78cb-jlsjf 0/2 Init:0/1
2 25m
demo-aai-aai-search-data-5d7fc5cc8c-hcrl9 2/2 Running
0 25m
demo-aai-aai-sparky-be-6598bf956c-6nc28 0/2 Init:0/1
0 25m
demo-aai-aai-spike-6546658f99-2bwr9 2/2 Running
0 25m
demo-aai-aai-traversal-5d8df67875-srngg 0/2 Init:0/1
2 25m
demo-aai-aai-traversal-update-query-data-xdf2w 0/1 Init:0/1
2 25m
This issue is blocking me from proceeding further. Any pointer is much
appreciated.
Thanks,
Thiriloshini
From: [email protected] <[email protected]>
Sent: Tuesday, March 19, 2019 7:58 AM
To: [email protected]; Thoppekrishnakumar, Thiriloshini
<[email protected]>
Cc: [email protected]; [email protected]; Rao, Ravi <[email protected]>
Subject: Re: [onap-discuss] [AAI] Changing dbedgerules in onap deployment
environment
Hi Thiriloshini,
in case it helps, regarding making new edgerules work in a running deployment
of Casablanca, this is what I did:
- add new edgerules in aai-resources, aai-traversal, aai-graphadmin, under v14
folder /opt/app/…/resources/schema/onap/dbedgerules/v14/
- run in aai-graphadmin as aaiadmin user the createDBSchema.sh script under
/opt/app/aai-graphadmin/bin/
- restart docker containers (not deleting the pod) aai-resources,
aai-traversal, aai-graphadmin
Regards
David
On 19 Mar 2019, at 05:04,
[email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>
wrote:
Keong,
I changed the EdgeRules in development environment by following the tutorial
and things work as expected.
- Changed the EdgeRule
- Build aai-common, resource, traversal, graphadmin.
- Started resource and traversal microservices as suggested in the
tutorial.
- Deleting esr-thirdparty-sdnc deleted the associated pnfs.
I build the docker images – traversal, resource, graphadmin from the source
code and used those images to run the container in
Deployment environment.
The dev-aai-graphadmin-create-db-schema fails. When I try to execute the
docker-entrypoint.sh and createDBSchema.sh manually inside the
dev-aai-graphadmin-create-db-schema container I see the below error
aaiadmin@dev-aai-graphadmin-create-db-schema-bh2b2:/opt/app/aai-graphadmin/scripts$
sh createDBSchema.sh
Tue Mar 19 02:54:55 UTC 2019 Starting createDBSchema.sh
---- NOTE --- about to open graph (takes a little while)--------;
-- Loading new schema elements into JanusGraph --
Failed to run the tool createDBSchema.sh successfully
I do not any errors in error.log under logs/createDBSchema
I also tried the below:
Instead of using the imaged build from development environment, used the same
images that comes with the Casablanca deployment environment. Changed the
edgerules in resource, traversal, graphadmin and run the createDBSchema.sh in
graphadmin. The edgerules change doesn’t take effect. I understand that this
method isn’t persistent.
Any pointer would be useful.
Thanks,
Thiriloshini
From: Keong Lim <[email protected]<mailto:[email protected]>>
Sent: Sunday, March 17, 2019 7:24 PM
To: Thoppekrishnakumar, Thiriloshini
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>; FORSYTH,
JAMES <[email protected]<mailto:[email protected]>>
Cc: Rao, Ravi <[email protected]<mailto:[email protected]>>
Subject: RE: [onap-discuss] [AAI] Changing dbedgerules in onap deployment
environment
Hi Thiriloshini,
Yes, the EdgeRules can be changed in a running OOM-based system, however the
aai-traversal is not enough.
As per the wiki Tutorial, the change also needs to be in schema-service,
graphadmin and aai-resources, and you need to run create-db-schema script to
apply it to the database.
However, the method you are using has not been documented, because it is not a
recommended method and it isn’t persistent when the pods are re-deployed.
If you can do it, please build new containers and deploy them into your
OOM-based system.
Keong
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]]
Sent: Saturday, 16 March 2019 18:10
To: [email protected]<mailto:[email protected]>; Keong Lim
<[email protected]<mailto:[email protected]>>; FORSYTH, JAMES
<[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>
Subject: [onap-discuss] [AAI] Changing dbedgerules in onap deployment
environment
Please find my questions below
1. Can the DbEdgeRules be changed in a deployment environment ?
2. If it can be done, am I changing in the right place ?
3. Please let me know if I am missing something here.
Any pointer is much appreciated.
Thanks,
Thiriloshini
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16214): https://lists.onap.org/g/onap-discuss/message/16214
Mute This Topic: https://lists.onap.org/mt/30451570/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Tue Mar 19 17:04:22 UTC 2019 Starting createDBSchema.sh
---- NOTE --- about to open graph (takes a little while)--------;
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at
org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:595)
Caused by: java.lang.ExceptionInInitializerError
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:103)
at org.onap.aai.schema.GenTester.main(GenTester.java:126)
... 8 more
Caused by: java.lang.RuntimeException: Failed to instantiate graphs
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:85)
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:57)
at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:90)
... 10 more
Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation
due to backend exception
at
org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at
org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at
org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.set(KCVSConfiguration.java:153)
at
org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.set(KCVSConfiguration.java:130)
at
org.janusgraph.diskstorage.configuration.ModifiableConfiguration.set(ModifiableConfiguration.java:40)
at
org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:163)
at
org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at
org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at
org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:115)
at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:82)
... 12 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not
successfully complete backend operation due to repeated temporary exceptions
after PT1M
at
org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at
org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 22 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary
failure in storage backend
at
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:413)
at
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.mutateMany(AstyanaxKeyColumnValueStore.java:280)
at
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.mutate(AstyanaxKeyColumnValueStore.java:276)
at
org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.mutate(ExpectedValueCheckingStore.java:79)
at
org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$2.call(KCVSConfiguration.java:158)
at
org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$2.call(KCVSConfiguration.java:153)
at
org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at
org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at
org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 23 more
Caused by:
com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException:
TokenRangeOfflineException: [host=10.42.7.194(10.42.7.194):9160, latency=0(0),
attempts=1]UnavailableException()
at
com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:28)
at
com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:153)
at
com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119)
at
com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:517)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:93)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:137)
at
org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:411)
... 31 more
Caused by: UnavailableException()
at
org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29615)
at
org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result$atomic_batch_mutate_resultStandardScheme.read(Cassandra.java:29592)
at
org.apache.cassandra.thrift.Cassandra$atomic_batch_mutate_result.read(Cassandra.java:29526)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_atomic_batch_mutate(Cassandra.java:1108)
at
org.apache.cassandra.thrift.Cassandra$Client.atomic_batch_mutate(Cassandra.java:1094)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:147)
at
com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1$1.internalExecute(ThriftKeyspaceImpl.java:141)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
... 39 more
Failed to run the tool createDBSchema.sh successfully
