Suraj, See my comments inline:
On Fri, Jul 11, 2014 at 12:51 AM, Suraj Nayak M <[email protected]> wrote: > This worked! :) > > Thanks a lot for your help Yusaku, Sid and Jeff! > > Just to understand why this happened I have few questions, so we can address > the issue if it is commonly occurring on machine failure thus reducing > manual intervention. (I will try to find answers for the below questions > myself in the source, would be great if I can get help here). > > Why did Postgresql corrupted on machine crash and did not recover on restart > ? You can read up on Postgres (or other databases) getting corrupt on machine shutdown, etc. For example, see http://rhaas.blogspot.com/2012/03/why-is-my-database-corrupted.html > If we had configured MySQL instead of Postgresql, will the problem re-occur > on similar situation ? (I know this may take a while to test and infer > conclusion) I'm no expert at making comments on MySQL vs Postgres, but for systems utilizing *any* RDBMS, it is recommended that backups are taken at an interval or (near) realtime through replication. > What is stored in key_value_store and which component uses it ? key_value_store is used only by Ambari Web UI to persist UI state. Ambari Server itself only exposes this feature but does not use it. Of course, any custom UI/client built upon the Ambari API could use it as well to persist client state. > Can we have this as a quick fix in UI when the Web UI pops up the error ? > Thus allowing admin to perform step by step guidance to recover from the > failure(manual or self healing process, need discussion on this). > What did the POST call to http://localhost:8080/api/v1/persist API do ? That is an interesting idea. The "magic" call to clear the key_store_value (/api/v1/persist) actually solves a number of issues, so it might be a good idea to expose this somehow. Yusaku > > -- > Suraj Nayak > > > On Friday 11 July 2014 05:17 AM, Yusaku Sako wrote: > > Please try the following (worked for me on my test cluster). > > 1. Delete all rows from key_value_store > 1. ambari-server restart > 2. Wait until the server comes up > 3. curl -i -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d > '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}' > http://localhost:8080/api/v1/persist > 4. Log back in to ambari-server > > Hope this helps. > Yusaku > > On Thu, Jul 10, 2014 at 11:31 AM, Suraj Nayak M <[email protected]> wrote: > > Thanks Sid, > > Found the username, password and database name used by ambari :) > > Yusaku, > Is it safe to truncate the table key_value_store table ? The old cluster > state will be restored after following steps mentioned below (confirmation > needed!): > > Truncate table key_value_store from postgresql. > Call the API as mentioned below in the email chain. > > -- > Suraj Nayak > > > On Thursday 10 July 2014 10:29 PM, Siddharth Wagle wrote: > > Hi Suraj, > > You should find all db info in /etc/ambari-server/conf/ambari.properties. > (grep for jdbc). > > I believe you can just truncate the table key_value_store and then try > making the API call. > Since UI utilizes this table Yusaku would be the right person to +1 this > advice. > > -Sid > > > On Thu, Jul 10, 2014 at 1:34 AM, Suraj Nayak M <[email protected]> wrote: > > Yes Sid, Even I suspected the Postgres corruption. > > I tried the API provided by Yasaku. I got 500 Internal error message > mentioned below : > > HTTP/1.1 500 Internal Exception: org.postgresql.util.PSQLException: > ERROR: missing chunk number 0 for toast value 50294 in pg_toast_16548 Error > Code: 0 Call: SELECT "key", "value" FROM key_value_store WHERE ("key" = ?) > bind => [1 parameter bound] Query: ReadObjectQuery(name="readObject" > referenceClass=KeyValueEntity sql="SELECT "key", "value" FROM > key_value_store WHERE ("key" = ?)") > Set-Cookie: AMBARISESSIONID=39gi1jtp0amp1orp572oipb8k;Path=/ > Content-Type: text/plain;charset=ISO-8859-1 > Content-Length: 467 > Server: Jetty(7.6.7.v20120910) > > { > "status": 500, > "message": "\nInternal Exception: org.postgresql.util.PSQLException: > ERROR: missing chunk number 0 for toast value 50294 in pg_toast_16548\nError > Code: 0\nCall: SELECT \"key\", \"value\" FROM key_value_store WHERE (\"key\" > \u003d ?)\n\tbind \u003d\u003e [1 parameter bound]\nQuery: > ReadObjectQuery(name\u003d\"readObject\" referenceClass\ > u003dKeyValueEntity > sql\u003d\"SELECT \"key\", \"value\" FROM key_value_store WHERE (\"key\" > \u003d ?)\")" > > I can notice a specific key, which is highlighter in red. Does deleting > this solve the problem ? > > I am not able to take a dump of existing databases as I don't know the > username and password of postgre. > > What is the default possource files at /var/lib/ambari-server/resources... > Server PID at: /var/run/ambari-server/ambari-server.pid > Server out at: /var/log/ambari-server/ambari-server.out > Server log at: /var/log/ambari-server/ambari-server.log > Ambari Server 'start' completed successfully. > [root@vm1 ~]# curl -i -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d > curl: option -d: requires parameter > curl: try 'curl --help' or 'curl --manual' for more information > [root@vm1 ~]# > '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}' > -bash: > {"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}: > command not found > [root@vm1 ~]# > '{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJotgres > username, password and database name that > Ambari uses ? > I suspect the user is posgres, which is the default user name. If I hard > reset the postgres which is mentioned in the this link, will ambari face > problem ? Which is the configuration file Ambari uses to store postgres > username and password? > > -- > Suraj Nayak > > > On Thursday 10 July 2014 09:31 AM, Yusaku Sako wrote: > > Suraj, > > Here's the API call that Sid mentioned for restoring key_value_store data: > > curl -i -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{ > "CLUSTER_CURRENT_STATUS": "{\"clusterState\":\"CLUSTER_STARTED_5\"}" > }' http://localhost:8080/api/v1/persist > > Hopefully this helps. > > Yusaku > > On Wed, Jul 9, 2014 at 8:49 PM, Siddharth Wagle <[email protected]> > wrote: > > Hi Suraj, > > Found a similar thread on postgres forum, > > http://postgresql.1045698.n5.nabble.com/select-table-indicate-missing-chunk-number-0-for-toast-value-96635-in-pg-toast-2619-td5682176.html > > The cleanup commands at the end of the topic seem worth a try. > > Note: If key_value_store has corrupt data it can be deleted without > affection state of the system, there is a API call to do it, Yusaku can > probably help you get the API call for it. > > 1. Backup the database as it is. > 2. I would try the API call and see if everything recovers. > 3. Run the analyze command and re index on pg_statistic. > > -Sid > > > On Wed, Jul 9, 2014 at 8:32 PM, Yusaku Sako <[email protected]> > wrote: > > "Caused by: org.postgresql.util.PSQLException: ERROR: missing chunk > number 0 for toast value 50294 in pg_toast_16548" > > I think this suggests that your Postgres DB is corrupt. > > Yusaku > > > On Wed, Jul 9, 2014 at 10:58 AM, Suraj Nayak M <[email protected]> wrote: > > Jeff, > > I tried restarting both ambari-server and postgresql. Still the error > exist. > I have attached a part of the log file(log after ambari-server restart) > so > it might help to pin point the issue. > > I could see following line before the Exception : > > 11:03:04,784 ERROR [qtp2062231465-43] HeartBeatHandler:127 - > CurrentResponseId unknown for vm1.somedomain.com - send register command > 11:03:09,016 ERROR [qtp2062231465-43] HeartBeatHandler:127 - > CurrentResponseId unknown for vm3.somedomain.com - send register command > 11:03:20,792 ERROR [qtp2062231465-44] HeartBeatHandler:127 - > CurrentResponseId unknown for vm2.somedomain.com - send register command > > Below are the output of the commands(Restart and status) : > > ------------------------------- > > Status : > > [root@vm1 ~]# ambari-server status > Using python /usr/bin/python2.6 > Ambari-server status > Ambari Server running > Found Ambari Server PID: '4895 at: > /var/run/ambari-server/ambari-server.pid > [root@vm1 ~]# ps -ef | grep postgres > postgres 1947 1 0 04:16 ? 00:00:01 /usr/bin/postmaster -p > 5432 > -D /var/lib/pgsql/data > postgres 1949 1947 0 04:16 ? 00:00:00 postgres: logger process > postgres 1951 1947 0 04:16 ? 00:00:03 postgres: writer process > postgres 1952 1947 0 04:16 ? 00:00:02 postgres: wal writer > process > postgres 1953 1947 0 04:16 ? 00:00:00 postgres: autovacuum > launcher process > postgres 1954 1947 0 04:16 ? 00:00:01 postgres: stats > collector > process > postgres 4915 1947 1 04:43 ? 00:05:05 postgres: ambari ambari > 127.0.0.1(57245) idle > postgres 5018 1947 0 04:44 ? 00:00:00 postgres: ambari ambari > 127.0.0.1(57267) idle > postgres 5173 1947 0 04:45 ? 00:00:00 postgres: ambari ambari > 127.0.0.1(57282) idle > postgres 5174 1947 0 04:45 ? 00:00:00 postgres: ambari ambari > 127.0.0.1(57283) idle > postgres 5175 1947 0 04:45 ? 00:00:00 postgres: ambari ambari > 127.0.0.1(57284) idle > root 30657 30493 0 10:59 pts/0 00:00:00 grep postgres > [root@vm1 ~]# service postgresql status > postmaster (pid 1947) is running... > > Restart : > [root@vm1 ~]# service postgresql restart > Stopping postgresql service: [ OK ] > Starting postgresql service: [ OK ] > [root@vm1 ~]# ambari-server restart > Using python /usr/bin/python2.6 > Restarting ambari-server > Using python /usr/bin/python2.6 > Stopping ambari-server > Ambari Server stopped > Using python /usr/bin/python2.6 > Starting ambari-server > Ambari Server running with 'root' privileges. > Organizing resource files at /var/lib/ambari-server/resources... > Server PID at: /var/run/ambari-server/ambari-server.pid > Server out at: /var/log/ambari-server/ambari-server.out > Server log at: /var/log/ambari-server/ambari-server.log > Ambari Server 'start' completed successfully. > > Status : > [root@vm1 ~]# service postgresql status > postmaster (pid 31003) is running... > > [root@vm1 ~]# ps -ef | grep postgres > postgres 31003 1 0 11:02 ? 00:00:00 /usr/bin/postmaster -p > 5432 > -D /var/lib/pgsql/data > postgres 31005 31003 0 11:02 ? 00:00:00 postgres: logger process > postgres 31007 31003 0 11:02 ? 00:00:00 postgres: writer process > postgres 31008 31003 0 11:02 ? 00:00:00 postgres: wal writer > process > postgres 31009 31003 0 11:02 ? 00:00:00 postgres: autovacuum > launcher process > postgres 31010 31003 0 11:02 ? 00:00:00 postgres: stats > collector > process > postgres 31079 31003 1 11:02 ? 00:00:03 postgres: ambari ambari > 127.0.0.1(59377) idle > postgres 31350 31003 0 11:05 ? 00:00:00 postgres: ambari ambari > 127.0.0.1(59417) idle > postgres 31351 31003 0 11:05 ? 00:00:00 postgres: ambari ambari > 127.0.0.1(59418) idle > postgres 31352 31003 0 11:05 ? 00:00:00 postgres: ambari ambari > 127.0.0.1(59419) idle > root 31501 30493 0 11:06 pts/0 00:00:00 grep postgres > > [root@vm1 ~]# ambari-server status > Using python /usr/bin/python2.6 > Ambari-server status > Ambari Server running > Found Ambari Server PID: '31061 at: > /var/run/ambari-server/ambari-server.pid > > ---------------------------- > > > -- > Suraj Nayak > > > On Wednesday 09 July 2014 07:15 PM, Jeff Sposetti wrote: > > Can you confirm ambari server is running? And stop/start as necessary? > > ambari-server status > Also, confirm postgres is running after you stop/start ambari-server? > > ps -ef | grep postgres > > service postgresql status > > > > On Wed, Jul 9, 2014 at 8:49 AM, Suraj Nayak <[email protected]> wrote: > > Hi, > > I have a 3 node cluster with HA enabled via Ambari. Accidentally the > power > supply to all the nodes was disrupted. After the power was restored, > the 3 > servers were turned on. Now if I try to login to Ambari Web UI, an > error > pops up Error in retrieving web client state from ambari server.(After > entering username and password) > > It doesn't show the cluster dashboard which was showing before. Instead > it > shows "Cluster Install Wizard" to setup a new cluster. How can I > recover > from the error? I do not want to restart the installation process > > Below is the error I found in /var/log/ambari-server/ambari-server.log > log > file. > > I found an email archive at > > > http://mail-archives.apache.org/mod_mbox/ambari-user/201402.mbox/%3ccf1bc3ed.2e43f%[email protected]%3E. > But I need suggestions on recovering but not starting the installation > from > scratch. > > Error : > > Local Exception Stack: > Exception [EclipseLink-4002] (Eclipse Persistence Services - > 2.4.0.v20120608-r11652): > org.eclipse.persistence.exceptions.DatabaseException > Internal Exception: org.postgresql.util.PSQLException: ERROR: missing > chunk number 0 for toast value 50294 in pg_toast_16548 > Error Code: 0 > Call: SELECT "key", "value" FROM key_value_store WHERE ("key" = ?) > bind => [1 parameter bound] > Query: ReadObjectQuery(name="readObject" referenceClass=KeyValueEntity > sql="SELECT "key", "value" FROM key_value_store WHERE ("key" = ?)") > at > > > org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333) > at > > > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:646) > at > > > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:537) > at > > > org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1800) > at > > > org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566) > at > > > org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207) > at > > > org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193) > at > > > org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:668) > at > > > org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2744) > at > > > org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2697) > at > > > org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:452) > at > > > org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1149) > at > > > org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852) > at > > > org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1108) > at > > > org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:420) > at > > > org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1196) > at > > > org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2875) > at > > > org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1602) > at > > > org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1584) > at > > > org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1535) > at > > > org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:838) > at > > > org.eclipse.persistence.internal.jpa.EntityManagerImpl.findInternal(EntityManagerImpl.java:778) > at > > > org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:671) > at > > > org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:543) > at > > > org.apache.ambari.server.orm.dao.KeyValueDAO.findByKey(KeyValueDAO.java:42) > at > > > org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:53) > at > > > org.apache.ambari.server.api.services.PersistKeyValueImpl.getValue(PersistKeyValueImpl.java:50) > at > > > org.apache.ambari.server.api.services.PersistKeyValueService.getKey(PersistKeyValueService.java:83) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > at > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > at > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > at > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > at > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > at > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652) > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > > > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > > > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.apache.ambari.server.security.authorization.internal.InternalTokenAuthenticationFilter.doFilter(InternalTokenAuthenticationFilter.java:53) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) > at > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > > > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) > at > > > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) > at > > > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) > at > > > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > at > > > org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:48) > at > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > at > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445) > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559) > at > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) > at > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038) > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374) > at > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) > at > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972) > at > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) > at > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:363) > at > > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483) > at > > > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920) > at > > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > > > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > > > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) > at > > > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.postgresql.util.PSQLException: ERROR: missing chunk > number > 0 for toast value 50294 in pg_toast_16548 > at > > > org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161) > at > > > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890) > at > > > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) > at > > > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559) > at > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) > at > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302) > at > > > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:938) > at > > > org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:609) > ... 98 more > > > -- > Thanks > Suraj Nayak > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the > reader of > this message is not the intended recipient, you are hereby notified that > any > printing, copying, dissemination, distribution, disclosure or forwarding > of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete > it > from your system. Thank You. > > > -- > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified > that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender > immediately > and delete it from your system. Thank You. > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of > this message is not the intended recipient, you are hereby notified that > any > printing, copying, dissemination, distribution, disclosure or forwarding > of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete > it > from your system. Thank You. > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete it > from your system. Thank You. > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
