Hi, our Oozie instance (managed by Cloudera Manager) get a
org.apache.openjpa.lib.jdbc.ReportingSQLException almost once a week.
When this happens, we see this Stacktraces in the log files:
2013-06-17 09:01:20,047 ERROR org.apache.oozie.command.wf.JobsXCommand:
USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Error,
2013-06-17 09:01:53,330 WARN org.apache.oozie.service.JPAService: USER[-]
GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] JPAExecutor
[WorkflowsJobGetJPAExecutor] ended with an active transaction, rolling back
2013-06-17 09:01:53,331 ERROR org.apache.oozie.command.wf.JobsXCommand:
USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] XException,
org.apache.oozie.command.CommandException: E0603: SQL error in operation,
Cannot issue commit in a nested connection when there is a pending
operation in the parent connection. {prepstmnt 320277519 SELECT t0.id,
t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
at org.apache.oozie.command.wf.JobsXCommand.execute(
JobsXCommand.java:72)
at org.apache.oozie.command.wf.JobsXCommand.execute(
JobsXCommand.java:32)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.DagEngine.getJobs(DagEngine.java:439)
at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs(
V1JobsServlet.java:314)
at org.apache.oozie.servlet.V1JobsServlet.getJobs(
V1JobsServlet.java:141)
at org.apache.oozie.servlet.BaseJobsServlet.doGet(
BaseJobsServlet.java:121)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at org.apache.oozie.servlet.JsonRestServlet.service(
JsonRestServlet.java:286)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(
AuthFilter.java:126)
at org.apache.hadoop.security.authentication.server.
AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(
HostnameFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(
Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.
process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(
JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: <openjpa-2.1.0-r422266:1071316 fatal general error>
org.apache.openjpa.persistence.PersistenceException: Cannot issue commit in
a nested connection when there is a pending operation in the parent
connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status,
t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time,
t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE
t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000,
state=X0Y66]
at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(
DBDictionary.java:4869)
at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(
DBDictionary.java:4829)
at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
SQLExceptions.java:136)
at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
SQLExceptions.java:110)
at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
SQLExceptions.java:62)
at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:139)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
at org.apache.openjpa.kernel.DelegatingQuery.execute(
DelegatingQuery.java:542)
at org.apache.openjpa.persistence.QueryImpl.execute(
QueryImpl.java:305)
at org.apache.openjpa.persistence.QueryImpl.
getResultList(QueryImpl.java:319)
at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
WorkflowsJobGetJPAExecutor.java:247)
at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
WorkflowsJobGetJPAExecutor.java:40)
at org.apache.oozie.service.JPAService.execute(JPAService.java:211)
at org.apache.oozie.command.wf.JobsXCommand.execute(
JobsXCommand.java:61)
... 29 more
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Cannot issue
commit in a nested connection when there is a pending operation in the
parent connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name,
t0.status, t0.run, t0.user_name, t0.group_name, t0.created_time,
t0.start_time, t0.last_modified_time, t0.end_time, t0.external_id FROM
WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]}
[code=30000, state=X0Y66]
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
LoggingConnectionDecorator.java:281)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
LoggingConnectionDecorator.java:265)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.
access$700(LoggingConnectionDecorator.java:72)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$
LoggingConnection$LoggingPreparedStatement.executeQuery(
LoggingConnectionDecorator.java:1183)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
executeQuery(DelegatingPreparedStatement.java:284)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$
CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1785)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
executeQuery(DelegatingPreparedStatement.java:274)
at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:118)
... 39 more
2013-06-17 09:01:54,275 WARN org.apache.oozie.servlet.V1JobsServlet:
USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[GET
http://ip-10-0-1-149.eu-west-1.compute.internal:11000/oozie/v1/jobs?len=100&doAs=hue&filter=&user.name=hue&timezone=America%2FLos_Angeles&jobtype=wf]
error[E0603], E0603: SQL error in operation, Cannot issue commit in a
nested connection when there is a pending operation in the parent
connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status,
t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time,
t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE
t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000,
state=X0Y66]
org.apache.oozie.servlet.XServletException: E0603: SQL error in operation,
Cannot issue commit in a nested connection when there is a pending
operation in the parent connection. {prepstmnt 320277519 SELECTt0.id,
t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs(
V1JobsServlet.java:323)
at org.apache.oozie.servlet.V1JobsServlet.getJobs(
V1JobsServlet.java:141)
at org.apache.oozie.servlet.BaseJobsServlet.doGet(
BaseJobsServlet.java:121)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at org.apache.oozie.servlet.JsonRestServlet.service(
JsonRestServlet.java:286)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(
AuthFilter.java:126)
at org.apache.hadoop.security.authentication.server.
AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(
HostnameFilter.java:84)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(
Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.
process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(
JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.DagEngineException: E0603: SQL error in
operation, Cannot issue commit in a nested connection when there is a
pending operation in the parent connection. {prepstmnt 320277519 SELECT
t0.id, t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
at org.apache.oozie.DagEngine.getJobs(DagEngine.java:442)
at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs(
V1JobsServlet.java:314)
... 25 more
Caused by: org.apache.oozie.command.CommandException: E0603: SQL error in
operation, Cannot issue commit in a nested connection when there is a
pending operation in the parent connection. {prepstmnt 320277519 SELECT
t0.id, t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
at org.apache.oozie.command.wf.JobsXCommand.execute(
JobsXCommand.java:72)
at org.apache.oozie.command.wf.JobsXCommand.execute(
JobsXCommand.java:32)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.DagEngine.getJobs(DagEngine.java:439)
... 26 more
Caused by: <openjpa-2.1.0-r422266:1071316 fatal general error>
org.apache.openjpa.persistence.PersistenceException: Cannot issue commit in
a nested connection when there is a pending operation in the parent
connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status,
t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time,
t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE
t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000,
state=X0Y66]
at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(
DBDictionary.java:4869)
at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(
DBDictionary.java:4829)
at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
SQLExceptions.java:136)
at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
SQLExceptions.java:110)
at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
SQLExceptions.java:62)
at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:139)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
at org.apache.openjpa.kernel.DelegatingQuery.execute(
DelegatingQuery.java:542)
at org.apache.openjpa.persistence.QueryImpl.execute(
QueryImpl.java:305)
at org.apache.openjpa.persistence.QueryImpl.
getResultList(QueryImpl.java:319)
at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
WorkflowsJobGetJPAExecutor.java:247)
at org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
WorkflowsJobGetJPAExecutor.java:40)
at org.apache.oozie.service.JPAService.execute(JPAService.java:211)
at org.apache.oozie.command.wf.JobsXCommand.execute(
JobsXCommand.java:61)
... 29 more
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Cannot issue
commit in a nested connection when there is a pending operation in the
parent connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name,
t0.status, t0.run, t0.user_name, t0.group_name, t0.created_time,
t0.start_time, t0.last_modified_time, t0.end_time, t0.external_id FROM
WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]}
[code=30000, state=X0Y66]
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
LoggingConnectionDecorator.java:281)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
LoggingConnectionDecorator.java:265)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.
access$700(LoggingConnectionDecorator.java:72)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$
LoggingConnection$LoggingPreparedStatement.executeQuery(
LoggingConnectionDecorator.java:1183)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
executeQuery(DelegatingPreparedStatement.java:284)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$
CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1785)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
executeQuery(DelegatingPreparedStatement.java:274)
at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:118)
... 39 more
After resting the service, the systems will run fine.
I have no idea where to debug or how to solve this.
Best Regards,
Christian.
