Hi Braden,
Thanks for the suggestions. When the dba killed the Oracle session a stack
trace was generated on the app server, and is shown below [1].
Will update the list as more information comes to hand.
Regards,
Mark.
[1] stack trace generated on app server after dba killed the oracle db session.
30.07.2012 15:07:17.404 *WARN* [10.9.74.20 [1343610835725] POST /p/mvk4oOC0aa
HTTP/1.1] org.sakaiproject.nakamura.lite.storage.jdbc.JDBCStorageClient Failed
to perform insert/update operation on [n, cn,
XWyycNndEeG6rgufCgRfQw+]:java.sql.SQLRecoverableException: ORA-00028: your
session has been killed
ORA-00028: your session has been killed
:{}
30.07.2012 15:07:17.405 *ERROR* [10.9.74.20 [1343610835725] POST /p/mvk4oOC0aa
HTTP/1.1]
org.sakaiproject.nakamura.resource.lite.servlet.post.operations.ImportOperation
Exception during response processing.
org.sakaiproject.nakamura.api.lite.StorageClientException: ORA-00028: your
session has been killed
ORA-00028: your session has been killed
at
org.sakaiproject.nakamura.lite.storage.jdbc.JDBCStorageClient.insert(JDBCStorageClient.java:439)
at
org.sakaiproject.nakamura.lite.CachingManager.putCached(CachingManager.java:181)
at
org.sakaiproject.nakamura.lite.content.ContentManagerImpl.update(ContentManagerImpl.java:542)
at
org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.internalImportContent(LiteJsonImporter.java:247)
at
org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.internalImportContent(LiteJsonImporter.java:205)
at
org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.internalImportContent(LiteJsonImporter.java:205)
at
org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.importContent(LiteJsonImporter.java:133)
at
org.sakaiproject.nakamura.resource.lite.servlet.post.operations.ImportOperation.doRun(ImportOperation.java:139)
at
org.sakaiproject.nakamura.api.resource.lite.AbstractSparsePostOperation.run(AbstractSparsePostOperation.java:102)
at
org.sakaiproject.nakamura.resource.lite.servlet.post.SparsePostServlet.doPost(SparsePostServlet.java:204)
at
org.apache.sling.api.servlets.SlingAllMethodsServlet.mayService(SlingAllMethodsServlet.java:148)
at
org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:344)
at
org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:375)
at
org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:491)
at
org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:273)
at
org.apache.sling.engine.impl.filter.RequestSlingFilterChain.render(RequestSlingFilterChain.java:49)
at
org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:64)
at
org.sakaiproject.nakamura.batch.RequestEventsFilter.doFilter(RequestEventsFilter.java:96)
at
org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
at
org.sakaiproject.nakamura.files.migrator.FileMigratorFilter.doFilter(FileMigratorFilter.java:82)
at
org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
at
org.sakaiproject.nakamura.http.usercontent.UserContentFilter.doFilter(UserContentFilter.java:89)
at
org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
at
org.sakaiproject.nakamura.cluster.ClusterTrackingFilter.doFilter(ClusterTrackingFilter.java:86)
at
org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
at
org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter.doFilter(RequestProgressTrackerLogFilter.java:59)
at
org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
at
org.apache.sling.engine.impl.SlingRequestProcessorImpl.processRequest(SlingRequestProcessorImpl.java:163)
at
org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:187)
at
org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:96)
at
org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:79)
at
org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42)
at
org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:49)
at
org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at
org.sakaiproject.nakamura.lite.http.SparseSessionTrackerImpl.doFilter(SparseSessionTrackerImpl.java:78)
at
org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at
org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at
org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at
org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at
org.sakaiproject.nakamura.http.cache.CacheControlFilter.doFilter(CacheControlFilter.java:190)
at
org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at
org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at
org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at
org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at
org.sakaiproject.nakamura.http.usercontent.SafeHostFilter.doFilter(SafeHostFilter.java:93)
at
org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at
org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at
org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at
org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at
org.sakaiproject.nakamura.files.migrator.FileMigratorFilter.doFilter(FileMigratorFilter.java:82)
at
org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at
org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at
org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at
org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at
org.sakaiproject.nakamura.http.qos.QoSFilter.doFilter(QoSFilter.java:165)
at
org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88)
at
org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76)
at
org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47)
at
org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33)
at
org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48)
at
org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39)
at
org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.sql.SQLRecoverableException: ORA-00028: your session has been
killed
ORA-00028: your session has been killed
at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:765)
at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:662)
at
oracle.jdbc.driver.T2CPreparedStatement.executeForDescribe(T2CPreparedStatement.java:546)
at
oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:729)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
at
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at
oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665)
at
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352)
at
org.sakaiproject.nakamura.lite.storage.jdbc.JDBCStorageClient.insert(JDBCStorageClient.java:398)
... 75 more
30.07.2012 15:07:17.406 *INFO* [10.9.74.20 [1343610835725] POST /p/mvk4oOC0aa
HTTP/1.1] logs/request.log 30/Jul/2012:15:07:17 +1000 [72178] <- 500 text/html
14001681ms
-----Original Message-----
From: Branden Visser [mailto:[email protected]]
Sent: Tuesday, 31 July 2012 9:57 PM
To: Walsh, Mark
Cc: [email protected]
Subject: Re: [oae-dev] OAE 1.3 and Oracle row level contention.
Hi Mark,
Was there a thread on the app server associated to the connection? If
there was, there may have been a stack-trace in the app server logs
when the DBA killed the session, that could help track down the
problem. If it happens again, taking a thread dump of the app server
before the DB session is killed might also help track it down. If you
get more information, please post it here.
Cheers,
Branden
On Tue, Jul 31, 2012 at 6:00 AM, Walsh, Mark <[email protected]> wrote:
> Hi,
>
> We are running OAE 1.3, and our systems/dba people are seeing row level
> contention in our Oracle database, in table cn_css_b. It appears a orphaned
> transaction/connection holds a row lock for hours, until our dba kills that
> database session.
>
> Just wondering if anyone else has experienced a similar problem.
>
> Mark.
> Charles Sturt University
>
> | ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN | MELBOURNE |
> ONTARIO | ORANGE | PORT MACQUARIE | SYDNEY | WAGGA WAGGA |
>
> LEGAL NOTICE
> This email (and any attachment) is confidential and is intended for the use
> of the addressee(s) only. If you are not the intended recipient of this
> email, you must not copy, distribute, take any action in reliance on it or
> disclose it to anyone. Any confidentiality is not waived or lost by reason of
> mistaken delivery. Email should be checked for viruses and defects before
> opening. Charles Sturt University (CSU) does not accept liability for viruses
> or any consequence which arise as a result of this email transmission. Email
> communications with CSU may be subject to automated email filtering, which
> could result in the delay or deletion of a legitimate email before it is read
> at CSU. The views expressed in this email are not necessarily those of CSU.
>
> Charles Sturt University in Australia http://www.csu.edu.au The Grange
> Chancellery, Panorama Avenue, Bathurst NSW Australia 2795 (ABN: 83 878 708
> 551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC), 02960B (ACT)).
> TEQSA Provider Number: PV12018
>
> Charles Sturt University in Ontario http://www.charlessturt.ca 860
> Harrington Court, Burlington Ontario Canada L7N 3N4 Registration:
> www.peqab.ca
>
> Consider the environment before printing this email.
> _______________________________________________
> oae-dev mailing list
> [email protected]
> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
--
Cheers,
Branden
Charles Sturt University
| ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN | MELBOURNE | ONTARIO
| ORANGE | PORT MACQUARIE | SYDNEY | WAGGA WAGGA |
LEGAL NOTICE
This email (and any attachment) is confidential and is intended for the use of
the addressee(s) only. If you are not the intended recipient of this email, you
must not copy, distribute, take any action in reliance on it or disclose it to
anyone. Any confidentiality is not waived or lost by reason of mistaken
delivery. Email should be checked for viruses and defects before opening.
Charles Sturt University (CSU) does not accept liability for viruses or any
consequence which arise as a result of this email transmission. Email
communications with CSU may be subject to automated email filtering, which
could result in the delay or deletion of a legitimate email before it is read
at CSU. The views expressed in this email are not necessarily those of CSU.
Charles Sturt University in Australia http://www.csu.edu.au The Grange
Chancellery, Panorama Avenue, Bathurst NSW Australia 2795 (ABN: 83 878 708
551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC), 02960B (ACT)). TEQSA
Provider Number: PV12018
Charles Sturt University in Ontario http://www.charlessturt.ca 860 Harrington
Court, Burlington Ontario Canada L7N 3N4 Registration: www.peqab.ca
Consider the environment before printing this email.
_______________________________________________
oae-dev mailing list
[email protected]
http://collab.sakaiproject.org/mailman/listinfo/oae-dev
