Can you confirm ambari server is running? And stop/start as necessary? ambari-server status
Also, confirm postgres is running after you stop/start ambari-server? ps -ef | grep postgres service postgresql status On Wed, Jul 9, 2014 at 8:49 AM, Suraj Nayak <[email protected]> wrote: > Hi, > > I have a 3 node cluster with HA enabled via Ambari. Accidentally the power > supply to all the nodes was disrupted. After the power was restored, the 3 > servers were turned on. Now if I try to login to Ambari Web UI, an error > pops up *Error in retrieving web client state from ambari server.*(After > entering username and password) > > It doesn't show the cluster dashboard which was showing before. Instead it > shows *"Cluster Install Wizard" *to setup a new cluster. How can I > recover from the error? I do not want to restart the installation process > > Below is the error I found in */var/log/ambari-server/ambari-server.log *log > file. > > I found an email archive at > http://mail-archives.apache.org/mod_mbox/ambari-user/201402.mbox/%3ccf1bc3ed.2e43f%[email protected]%3E. > But I need suggestions on recovering but not starting the installation from > scratch. > > *Error :* > > Local Exception Stack: > Exception [EclipseLink-4002] (Eclipse Persistence Services - > 2.4.0.v20120608-r11652): > org.eclipse.persistence.exceptions.DatabaseException > Internal Exception: org.postgresql.util.PSQLException: ERROR: missing > chunk number 0 for toast value 50294 in pg_toast_16548 > Error Code: 0 > Call: SELECT "key", "value" FROM key_value_store WHERE ("key" = ?) > bind => [1 parameter bound] > Query: ReadObjectQuery(name="readObject" referenceClass=KeyValueEntity > sql="SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)") > at > org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333) > at > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:646) > at > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:537) > at > org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1800) > at > org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566) > at > org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207) > at > org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193) > at > org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:668) > at > org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2744) > at > org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2697) > at > org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:452) > at > org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1149) > at > org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852) > at > org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1108) > at > org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:420) > at > org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1196) > at > org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2875) > at > org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1602) > at > org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1584) > at > org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1535) > at > org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:838) > at > org.eclipse.persistence.internal.jpa.EntityManagerImpl.findInternal(EntityManagerImpl.java:778) > at > org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:671) > at > org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:543) > at > org.apache.ambari.server.orm.dao.KeyValueDAO.findByKey(KeyValueDAO.java:42) > at > org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53) > at > org.apache.ambari.server.api.services.PersistKeyValueImpl.getValue(PersistKeyValueImpl.java:50) > at > org.apache.ambari.server.api.services.PersistKeyValueService.getKey(PersistKeyValueService.java:83) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > at > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > at > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > at > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.apache.ambari.server.security.authorization.internal.InternalTokenAuthenticationFilter.doFilter(InternalTokenAuthenticationFilter.java:53) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) > at > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) > at > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) > at > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > at > org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:48) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:363) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.postgresql.util.PSQLException: ERROR: missing chunk number > 0 for toast value 50294 in pg_toast_16548 > at > org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) > at > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) > at > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) > at > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559) > at > org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) > at > org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302) > at > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:938) > at > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:609) > ... 98 more > > > -- > Thanks > Suraj Nayak > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
