[
https://issues.apache.org/jira/browse/ATLAS-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108969#comment-16108969
]
Endre Kovacs commented on ATLAS-1903:
-------------------------------------
Hi,
Atlas 0.8-incubating uses Titan 0.5.4 under the hood,
which, when the commit to the secondary persistence fails, it does not add
information about the failure to the transaction, it only logs the failure to
the txLog and to the standard log.
https://github.com/thinkaurelius/titan/blob/0.5.4/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/database/StandardTitanGraph.java#L715
{code:java}
try {
//2. Commit indexes - [FAILURE] all exceptions are
collected and logged but nothing is aborted
indexFailures = mutator.commitIndexes();
if (!indexFailures.isEmpty()) {
status = LogTxStatus.SECONDARY_FAILURE;
for (Map.Entry<String,Throwable> entry :
indexFailures.entrySet()) {
log.error("Error while commiting index
mutations for transaction ["+transactionId+"] on index: "
+entry.getKey(),entry.getValue());
}
}
...
...
...
} finally {
if (logTransaction) {
//[FAILURE] An exception here will be logged and
not escalated; tx considered success and
// needs to be cleaned up later
try {
txLog.add(txLogHeader.serializeSecondary(serializer,status,indexFailures,userlogSuccess),txLogHeader.getLogKey());
} catch (Throwable e) {
log.error("Could not tx-log secondary
persistence status on transaction ["+transactionId+"]",e);
}
}
}
{code}
So the only feasible way to find out if the secondary persistence ran into
commit failure, if we register a read handle to that txLog and monitor for
failure messages there.
> When Solr is not available, REST API type update returns HTTP 200 OK although
> the update was not successful
> -----------------------------------------------------------------------------------------------------------
>
> Key: ATLAS-1903
> URL: https://issues.apache.org/jira/browse/ATLAS-1903
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 0.8-incubating
> Reporter: Péter Gergő Barna
> Assignee: Péter Gergő Barna
>
> After Solr is stopped, REST API type update is called:
> PUT http://127.0.0.1:21000/api/atlas/v2/types/typedefs
> It takes a long time to respond (3 mins). Finally it returns HTTP 200 OK, and
> a message body that suggests that the update was successful.
> However, the update was not successful, and from the logs it can be seen that
> the problem is Solr being unavailable
> I think, instead of HTTP 200 OK, the response should be HTTP 500 or 503 and a
> meaningful error message indicating that the problem is Solr being
> unavailable.
> Log excerpt:
> {noformat}
> 2017-06-28 15:36:52,747 ERROR - [pool-1-thread-8 -
> /api/atlas/v2/types/typedefs - c6d49838-c0d2-465a-93ce-38e63511fca5:] ~ Error
> while commiting index mutations for transaction [2] on index: search
> (StandardTitanGraph:688)
> com.thinkaurelius.titan.core.TitanException: Could not execute operation due
> to backend exception
> at
> com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
> at
> com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:119)
> at
> com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:100)
> at
> com.thinkaurelius.titan.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:124)
> at
> com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:684)
> at
> com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1337)
> at
> com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:60)
> at
> org.apache.atlas.repository.graphdb.titan0.Titan0Graph.commit(Titan0Graph.java:154)
> at
> org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:61)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
> at
> org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1$$EnhancerBySpringCGLIB$$f4b1ae3b.updateTypesDef(<generated>)
> at
> org.apache.atlas.web.rest.TypesREST.updateAtlasTypeDefs(TypesREST.java:326)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
> at
> org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:76)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.apache.atlas.web.filters.AtlasAuthorizationFilter.doFilter(AtlasAuthorizationFilter.java:157)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235)
> at
> org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
> at
> org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:325)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.apache.atlas.web.filters.AtlasKnoxSSOAuthenticationFilter.doFilter(AtlasKnoxSSOAuthenticationFilter.java:132)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.apache.atlas.web.filters.StaleTransactionCleanupFilter.doFilter(StaleTransactionCleanupFilter.java:55)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
> at
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException:
> Could not successfully complete backend operation due to repeated temporary
> exceptions after Duration[100 s]
> at
> com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:86)
> at
> com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
> ... 96 more
> Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException:
> Unable to complete query on Solr.
> at
> com.thinkaurelius.titan.diskstorage.solr.Solr5Index.storageException(Solr5Index.java:892)
> at
> com.thinkaurelius.titan.diskstorage.solr.Solr5Index.mutate(Solr5Index.java:354)
> at
> com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:122)
> at
> com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:119)
> at
> com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
> ... 97 more
> Caused by: org.apache.solr.common.SolrException: Could not load collection
> from ZK: vertex_index
> at
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:992)
> at
> org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:525)
> at
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:189)
> at
> org.apache.solr.common.cloud.ClusterState.hasCollection(ClusterState.java:119)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1111)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:833)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
> at
> com.thinkaurelius.titan.diskstorage.solr.Solr5Index.commitDocumentChanges(Solr5Index.java:435)
> at
> com.thinkaurelius.titan.diskstorage.solr.Solr5Index.mutate(Solr5Index.java:351)
> ... 100 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /collections/vertex_index/state.json
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
> at
> org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1004)
> at
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:990)
> ... 108 more
> 2017-06-28 15:36:52,748 DEBUG - [pool-1-thread-8 -
> /api/atlas/v2/types/typedefs - c6d49838-c0d2-465a-93ce-38e63511fca5:] ~ graph
> commit (GraphTransactionInterceptor:65)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
