Hi Christian, What version of Oozie are you using here? And what DB is it running against?
On Mon, Jun 17, 2013 at 4:18 PM, Christian Schneider <[email protected]> wrote: > Hi, our Oozie instance (managed by Cloudera Manager) get a > org.apache.openjpa.lib.jdbc.ReportingSQLException almost once a week. > > When this happens, we see this Stacktraces in the log files: > > 2013-06-17 09:01:20,047 ERROR org.apache.oozie.command.wf.JobsXCommand: > USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Error, > 2013-06-17 09:01:53,330 WARN org.apache.oozie.service.JPAService: USER[-] > GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] JPAExecutor > [WorkflowsJobGetJPAExecutor] ended with an active transaction, rolling back > 2013-06-17 09:01:53,331 ERROR org.apache.oozie.command.wf.JobsXCommand: > USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] XException, > org.apache.oozie.command.CommandException: E0603: SQL error in operation, > Cannot issue commit in a nested connection when there is a pending > operation in the parent connection. {prepstmnt 320277519 SELECT t0.id, > t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name, > t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time, > t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY > t0.created_time DESC [params=?]} [code=30000, state=X0Y66] > at org.apache.oozie.command.wf.JobsXCommand.execute( > JobsXCommand.java:72) > at org.apache.oozie.command.wf.JobsXCommand.execute( > JobsXCommand.java:32) > at org.apache.oozie.command.XCommand.call(XCommand.java:277) > at org.apache.oozie.DagEngine.getJobs(DagEngine.java:439) > at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs( > V1JobsServlet.java:314) > at org.apache.oozie.servlet.V1JobsServlet.getJobs( > V1JobsServlet.java:141) > at org.apache.oozie.servlet.BaseJobsServlet.doGet( > BaseJobsServlet.java:121) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at org.apache.oozie.servlet.JsonRestServlet.service( > JsonRestServlet.java:286) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > ApplicationFilterChain.java:290) > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > ApplicationFilterChain.java:206) > at org.apache.oozie.servlet.AuthFilter$2.doFilter( > AuthFilter.java:126) > at org.apache.hadoop.security.authentication.server. > AuthenticationFilter.doFilter(AuthenticationFilter.java:384) > at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > ApplicationFilterChain.java:235) > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > ApplicationFilterChain.java:206) > at org.apache.oozie.servlet.HostnameFilter.doFilter( > HostnameFilter.java:84) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > ApplicationFilterChain.java:235) > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > ApplicationFilterChain.java:206) > at org.apache.catalina.core.StandardWrapperValve.invoke( > StandardWrapperValve.java:233) > at org.apache.catalina.core.StandardContextValve.invoke( > StandardContextValve.java:191) > at org.apache.catalina.core.StandardHostValve.invoke( > StandardHostValve.java:127) > at org.apache.catalina.valves.ErrorReportValve.invoke( > ErrorReportValve.java:102) > at org.apache.catalina.core.StandardEngineValve.invoke( > StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service( > CoyoteAdapter.java:293) > at org.apache.coyote.http11.Http11Processor.process( > Http11Processor.java:859) > at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler. > process(Http11Protocol.java:602) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run( > JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:662) > Caused by: <openjpa-2.1.0-r422266:1071316 fatal general error> > org.apache.openjpa.persistence.PersistenceException: Cannot issue commit in > a nested connection when there is a pending operation in the parent > connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status, > t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time, > t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE > t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000, > state=X0Y66] > at org.apache.openjpa.jdbc.sql.DBDictionary.narrow( > DBDictionary.java:4869) > at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException( > DBDictionary.java:4829) > at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore( > SQLExceptions.java:136) > at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore( > SQLExceptions.java:110) > at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore( > SQLExceptions.java:62) > at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$ > PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:139) > at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005) > at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863) > at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794) > at org.apache.openjpa.kernel.DelegatingQuery.execute( > DelegatingQuery.java:542) > at org.apache.openjpa.persistence.QueryImpl.execute( > QueryImpl.java:305) > at org.apache.openjpa.persistence.QueryImpl. > getResultList(QueryImpl.java:319) > at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute( > WorkflowsJobGetJPAExecutor.java:247) > at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute( > WorkflowsJobGetJPAExecutor.java:40) > at org.apache.oozie.service.JPAService.execute(JPAService.java:211) > at org.apache.oozie.command.wf.JobsXCommand.execute( > JobsXCommand.java:61) > ... 29 more > Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Cannot issue > commit in a nested connection when there is a pending operation in the > parent connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, > t0.status, t0.run, t0.user_name, t0.group_name, t0.created_time, > t0.start_time, t0.last_modified_time, t0.end_time, t0.external_id FROM > WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} > [code=30000, state=X0Y66] > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap( > LoggingConnectionDecorator.java:281) > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap( > LoggingConnectionDecorator.java:265) > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator. > access$700(LoggingConnectionDecorator.java:72) > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$ > LoggingConnection$LoggingPreparedStatement.executeQuery( > LoggingConnectionDecorator.java:1183) > at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement. > executeQuery(DelegatingPreparedStatement.java:284) > at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$ > CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1785) > at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement. > executeQuery(DelegatingPreparedStatement.java:274) > at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$ > PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:118) > ... 39 more > 2013-06-17 09:01:54,275 WARN org.apache.oozie.servlet.V1JobsServlet: > USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[GET > http://ip-10-0-1-149.eu-west-1.compute.internal:11000/oozie/v1/jobs?len=100&doAs=hue&filter=&user.name=hue&timezone=America%2FLos_Angeles&jobtype=wf] > error[E0603], E0603: SQL error in operation, Cannot issue commit in a > nested connection when there is a pending operation in the parent > connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status, > t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time, > t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE > t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000, > state=X0Y66] > org.apache.oozie.servlet.XServletException: E0603: SQL error in operation, > Cannot issue commit in a nested connection when there is a pending > operation in the parent connection. {prepstmnt 320277519 SELECTt0.id, > t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name, > t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time, > t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY > t0.created_time DESC [params=?]} [code=30000, state=X0Y66] > at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs( > V1JobsServlet.java:323) > at org.apache.oozie.servlet.V1JobsServlet.getJobs( > V1JobsServlet.java:141) > at org.apache.oozie.servlet.BaseJobsServlet.doGet( > BaseJobsServlet.java:121) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at org.apache.oozie.servlet.JsonRestServlet.service( > JsonRestServlet.java:286) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > ApplicationFilterChain.java:290) > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > ApplicationFilterChain.java:206) > at org.apache.oozie.servlet.AuthFilter$2.doFilter( > AuthFilter.java:126) > at org.apache.hadoop.security.authentication.server. > AuthenticationFilter.doFilter(AuthenticationFilter.java:384) > at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > ApplicationFilterChain.java:235) > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > ApplicationFilterChain.java:206) > at org.apache.oozie.servlet.HostnameFilter.doFilter( > HostnameFilter.java:84) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > ApplicationFilterChain.java:235) > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > ApplicationFilterChain.java:206) > at org.apache.catalina.core.StandardWrapperValve.invoke( > StandardWrapperValve.java:233) > at org.apache.catalina.core.StandardContextValve.invoke( > StandardContextValve.java:191) > at org.apache.catalina.core.StandardHostValve.invoke( > StandardHostValve.java:127) > at org.apache.catalina.valves.ErrorReportValve.invoke( > ErrorReportValve.java:102) > at org.apache.catalina.core.StandardEngineValve.invoke( > StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service( > CoyoteAdapter.java:293) > at org.apache.coyote.http11.Http11Processor.process( > Http11Processor.java:859) > at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler. > process(Http11Protocol.java:602) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run( > JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.oozie.DagEngineException: E0603: SQL error in > operation, Cannot issue commit in a nested connection when there is a > pending operation in the parent connection. {prepstmnt 320277519 SELECT > t0.id, t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name, > t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time, > t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY > t0.created_time DESC [params=?]} [code=30000, state=X0Y66] > at org.apache.oozie.DagEngine.getJobs(DagEngine.java:442) > at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs( > V1JobsServlet.java:314) > ... 25 more > Caused by: org.apache.oozie.command.CommandException: E0603: SQL error in > operation, Cannot issue commit in a nested connection when there is a > pending operation in the parent connection. {prepstmnt 320277519 SELECT > t0.id, t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name, > t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time, > t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY > t0.created_time DESC [params=?]} [code=30000, state=X0Y66] > at org.apache.oozie.command.wf.JobsXCommand.execute( > JobsXCommand.java:72) > at org.apache.oozie.command.wf.JobsXCommand.execute( > JobsXCommand.java:32) > at org.apache.oozie.command.XCommand.call(XCommand.java:277) > at org.apache.oozie.DagEngine.getJobs(DagEngine.java:439) > ... 26 more > Caused by: <openjpa-2.1.0-r422266:1071316 fatal general error> > org.apache.openjpa.persistence.PersistenceException: Cannot issue commit in > a nested connection when there is a pending operation in the parent > connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status, > t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time, > t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE > t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000, > state=X0Y66] > at org.apache.openjpa.jdbc.sql.DBDictionary.narrow( > DBDictionary.java:4869) > at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException( > DBDictionary.java:4829) > at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore( > SQLExceptions.java:136) > at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore( > SQLExceptions.java:110) > at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore( > SQLExceptions.java:62) > at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$ > PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:139) > at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005) > at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863) > at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794) > at org.apache.openjpa.kernel.DelegatingQuery.execute( > DelegatingQuery.java:542) > at org.apache.openjpa.persistence.QueryImpl.execute( > QueryImpl.java:305) > at org.apache.openjpa.persistence.QueryImpl. > getResultList(QueryImpl.java:319) > at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute( > WorkflowsJobGetJPAExecutor.java:247) > at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute( > WorkflowsJobGetJPAExecutor.java:40) > at org.apache.oozie.service.JPAService.execute(JPAService.java:211) > at org.apache.oozie.command.wf.JobsXCommand.execute( > JobsXCommand.java:61) > ... 29 more > Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Cannot issue > commit in a nested connection when there is a pending operation in the > parent connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, > t0.status, t0.run, t0.user_name, t0.group_name, t0.created_time, > t0.start_time, t0.last_modified_time, t0.end_time, t0.external_id FROM > WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} > [code=30000, state=X0Y66] > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap( > LoggingConnectionDecorator.java:281) > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap( > LoggingConnectionDecorator.java:265) > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator. > access$700(LoggingConnectionDecorator.java:72) > at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$ > LoggingConnection$LoggingPreparedStatement.executeQuery( > LoggingConnectionDecorator.java:1183) > at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement. > executeQuery(DelegatingPreparedStatement.java:284) > at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$ > CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1785) > at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement. > executeQuery(DelegatingPreparedStatement.java:274) > at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$ > PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:118) > ... 39 more > > After resting the service, the systems will run fine. > > I have no idea where to debug or how to solve this. > > > Best Regards, > Christian. -- Harsh J
