Can you please add to the Jira how you reproduce the error. It would be great to have this replicated in either a Tsung or Jmeter test plan based on the recipe. We can then run this before releasing each tag.
Alan Alan Berg Group Education and Research Services Central Computer Services University of Amsterdam ________________________________________ From: [email protected] [[email protected]] on behalf of Walsh, Mark [[email protected]] Sent: 02 August 2012 05:47 To: 'Branden Visser' Cc: [email protected] Subject: Re: [oae-dev] OAE 1.3 and Oracle row level contention. Hi Branden, Thanks for looking into this. One of our systems guys has found a method to reproduce this problem. Stack trace and server logs have been sent to you. Regards, Mark. -----Original Message----- From: Branden Visser [mailto:[email protected]] Sent: Wednesday, 1 August 2012 11:22 PM To: Walsh, Mark Cc: [email protected] Subject: Re: [oae-dev] OAE 1.3 and Oracle row level contention. Hi Mark, thanks for the stack trace. Unfortunately, nothing jumps out at me. Can I get a wider set of app server log files to track down other exceptions? If it's feasible to zip them all up for me, that would be great. Thanks, Branden On Wed, Aug 1, 2012 at 1:51 AM, Walsh, Mark <[email protected]> wrote: > Hi Braden, > > Thanks for the suggestions. When the dba killed the Oracle session a stack > trace was generated on the app server, and is shown below [1]. > Will update the list as more information comes to hand. > > Regards, > Mark. > > [1] stack trace generated on app server after dba killed the oracle db > session. > 30.07.2012 15:07:17.404 *WARN* [10.9.74.20 [1343610835725] POST /p/mvk4oOC0aa > HTTP/1.1] org.sakaiproject.nakamura.lite.storage.jdbc.JDBCStorageClient > Failed to perform insert/update operation on [n, cn, > XWyycNndEeG6rgufCgRfQw+]:java.sql.SQLRecoverableException: ORA-00028: your > session has been killed > ORA-00028: your session has been killed > :{} > 30.07.2012 15:07:17.405 *ERROR* [10.9.74.20 [1343610835725] POST > /p/mvk4oOC0aa HTTP/1.1] > org.sakaiproject.nakamura.resource.lite.servlet.post.operations.ImportOperation > Exception during response processing. > org.sakaiproject.nakamura.api.lite.StorageClientException: ORA-00028: your > session has been killed > ORA-00028: your session has been killed > > at > org.sakaiproject.nakamura.lite.storage.jdbc.JDBCStorageClient.insert(JDBCStorageClient.java:439) > at > org.sakaiproject.nakamura.lite.CachingManager.putCached(CachingManager.java:181) > at > org.sakaiproject.nakamura.lite.content.ContentManagerImpl.update(ContentManagerImpl.java:542) > at > org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.internalImportContent(LiteJsonImporter.java:247) > at > org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.internalImportContent(LiteJsonImporter.java:205) > at > org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.internalImportContent(LiteJsonImporter.java:205) > at > org.sakaiproject.nakamura.api.resource.lite.LiteJsonImporter.importContent(LiteJsonImporter.java:133) > at > org.sakaiproject.nakamura.resource.lite.servlet.post.operations.ImportOperation.doRun(ImportOperation.java:139) > at > org.sakaiproject.nakamura.api.resource.lite.AbstractSparsePostOperation.run(AbstractSparsePostOperation.java:102) > at > org.sakaiproject.nakamura.resource.lite.servlet.post.SparsePostServlet.doPost(SparsePostServlet.java:204) > at > org.apache.sling.api.servlets.SlingAllMethodsServlet.mayService(SlingAllMethodsServlet.java:148) > at > org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:344) > at > org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:375) > at > org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:491) > at > org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:273) > at > org.apache.sling.engine.impl.filter.RequestSlingFilterChain.render(RequestSlingFilterChain.java:49) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:64) > at > org.sakaiproject.nakamura.batch.RequestEventsFilter.doFilter(RequestEventsFilter.java:96) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60) > at > org.sakaiproject.nakamura.files.migrator.FileMigratorFilter.doFilter(FileMigratorFilter.java:82) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60) > at > org.sakaiproject.nakamura.http.usercontent.UserContentFilter.doFilter(UserContentFilter.java:89) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60) > at > org.sakaiproject.nakamura.cluster.ClusterTrackingFilter.doFilter(ClusterTrackingFilter.java:86) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60) > at > org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter.doFilter(RequestProgressTrackerLogFilter.java:59) > at > org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60) > at > org.apache.sling.engine.impl.SlingRequestProcessorImpl.processRequest(SlingRequestProcessorImpl.java:163) > at > org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:187) > at > org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(ServletHandler.java:96) > at > org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:79) > at > org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(ServletPipeline.java:42) > at > org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:49) > at > org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > at > org.sakaiproject.nakamura.lite.http.SparseSessionTrackerImpl.doFilter(SparseSessionTrackerImpl.java:78) > at > org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88) > at > org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76) > at > org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47) > at > org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > at > org.sakaiproject.nakamura.http.cache.CacheControlFilter.doFilter(CacheControlFilter.java:190) > at > org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88) > at > org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76) > at > org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47) > at > org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > at > org.sakaiproject.nakamura.http.usercontent.SafeHostFilter.doFilter(SafeHostFilter.java:93) > at > org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88) > at > org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76) > at > org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47) > at > org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > at > org.sakaiproject.nakamura.files.migrator.FileMigratorFilter.doFilter(FileMigratorFilter.java:82) > at > org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88) > at > org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76) > at > org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47) > at > org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > at > org.sakaiproject.nakamura.http.qos.QoSFilter.doFilter(QoSFilter.java:165) > at > org.apache.felix.http.base.internal.handler.FilterHandler.doHandle(FilterHandler.java:88) > at > org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:76) > at > org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(InvocationFilterChain.java:47) > at > org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(HttpFilterChain.java:33) > at > org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(FilterPipeline.java:48) > at > org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:39) > at > org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:67) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Caused by: java.sql.SQLRecoverableException: ORA-00028: your session has been > killed > ORA-00028: your session has been killed > > at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:765) > at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:662) > at > oracle.jdbc.driver.T2CPreparedStatement.executeForDescribe(T2CPreparedStatement.java:546) > at > oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:729) > at > oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329) > at > oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584) > at > oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665) > at > oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352) > at > org.sakaiproject.nakamura.lite.storage.jdbc.JDBCStorageClient.insert(JDBCStorageClient.java:398) > ... 75 more > 30.07.2012 15:07:17.406 *INFO* [10.9.74.20 [1343610835725] POST /p/mvk4oOC0aa > HTTP/1.1] logs/request.log 30/Jul/2012:15:07:17 +1000 [72178] <- 500 > text/html 14001681ms > > -----Original Message----- > From: Branden Visser [mailto:[email protected]] > Sent: Tuesday, 31 July 2012 9:57 PM > To: Walsh, Mark > Cc: [email protected] > Subject: Re: [oae-dev] OAE 1.3 and Oracle row level contention. > > Hi Mark, > > Was there a thread on the app server associated to the connection? If > there was, there may have been a stack-trace in the app server logs > when the DBA killed the session, that could help track down the > problem. If it happens again, taking a thread dump of the app server > before the DB session is killed might also help track it down. If you > get more information, please post it here. > > Cheers, > Branden > > On Tue, Jul 31, 2012 at 6:00 AM, Walsh, Mark <[email protected]> wrote: >> Hi, >> >> We are running OAE 1.3, and our systems/dba people are seeing row level >> contention in our Oracle database, in table cn_css_b. It appears a orphaned >> transaction/connection holds a row lock for hours, until our dba kills that >> database session. >> >> Just wondering if anyone else has experienced a similar problem. >> >> Mark. >> Charles Sturt University >> >> | ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN | MELBOURNE | >> ONTARIO | ORANGE | PORT MACQUARIE | SYDNEY | WAGGA WAGGA | >> >> LEGAL NOTICE >> This email (and any attachment) is confidential and is intended for the use >> of the addressee(s) only. If you are not the intended recipient of this >> email, you must not copy, distribute, take any action in reliance on it or >> disclose it to anyone. Any confidentiality is not waived or lost by reason >> of mistaken delivery. Email should be checked for viruses and defects before >> opening. Charles Sturt University (CSU) does not accept liability for >> viruses or any consequence which arise as a result of this email >> transmission. Email communications with CSU may be subject to automated >> email filtering, which could result in the delay or deletion of a legitimate >> email before it is read at CSU. The views expressed in this email are not >> necessarily those of CSU. >> >> Charles Sturt University in Australia http://www.csu.edu.au The Grange >> Chancellery, Panorama Avenue, Bathurst NSW Australia 2795 (ABN: 83 878 708 >> 551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC), 02960B (ACT)). >> TEQSA Provider Number: PV12018 >> >> Charles Sturt University in Ontario http://www.charlessturt.ca 860 >> Harrington Court, Burlington Ontario Canada L7N 3N4 Registration: >> www.peqab.ca >> >> Consider the environment before printing this email. >> _______________________________________________ >> oae-dev mailing list >> [email protected] >> http://collab.sakaiproject.org/mailman/listinfo/oae-dev > > > > -- > Cheers, > Branden > Charles Sturt University > > | ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN | MELBOURNE | > ONTARIO | ORANGE | PORT MACQUARIE | SYDNEY | WAGGA WAGGA | > > LEGAL NOTICE > This email (and any attachment) is confidential and is intended for the use > of the addressee(s) only. If you are not the intended recipient of this > email, you must not copy, distribute, take any action in reliance on it or > disclose it to anyone. Any confidentiality is not waived or lost by reason of > mistaken delivery. Email should be checked for viruses and defects before > opening. Charles Sturt University (CSU) does not accept liability for viruses > or any consequence which arise as a result of this email transmission. Email > communications with CSU may be subject to automated email filtering, which > could result in the delay or deletion of a legitimate email before it is read > at CSU. The views expressed in this email are not necessarily those of CSU. > > Charles Sturt University in Australia http://www.csu.edu.au The Grange > Chancellery, Panorama Avenue, Bathurst NSW Australia 2795 (ABN: 83 878 708 > 551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC), 02960B (ACT)). > TEQSA Provider Number: PV12018 > > Charles Sturt University in Ontario http://www.charlessturt.ca 860 > Harrington Court, Burlington Ontario Canada L7N 3N4 Registration: > www.peqab.ca > > Consider the environment before printing this email. -- Cheers, Branden Charles Sturt University | ALBURY-WODONGA | BATHURST | CANBERRA | DUBBO | GOULBURN | MELBOURNE | ONTARIO | ORANGE | PORT MACQUARIE | SYDNEY | WAGGA WAGGA | LEGAL NOTICE This email (and any attachment) is confidential and is intended for the use of the addressee(s) only. If you are not the intended recipient of this email, you must not copy, distribute, take any action in reliance on it or disclose it to anyone. Any confidentiality is not waived or lost by reason of mistaken delivery. Email should be checked for viruses and defects before opening. Charles Sturt University (CSU) does not accept liability for viruses or any consequence which arise as a result of this email transmission. Email communications with CSU may be subject to automated email filtering, which could result in the delay or deletion of a legitimate email before it is read at CSU. The views expressed in this email are not necessarily those of CSU. Charles Sturt University in Australia http://www.csu.edu.au The Grange Chancellery, Panorama Avenue, Bathurst NSW Australia 2795 (ABN: 83 878 708 551; CRICOS Provider Numbers: 00005F (NSW), 01947G (VIC), 02960B (ACT)). TEQSA Provider Number: PV12018 Charles Sturt University in Ontario http://www.charlessturt.ca 860 Harrington Court, Burlington Ontario Canada L7N 3N4 Registration: www.peqab.ca Consider the environment before printing this email. _______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev _______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
