Hi Andras,
After switching to a MySQL database from Derby the jobs ran successfully for a
few days before the RollbackException returned again.
In terms of using the Oozie coordinator, will the workflows each need a
coordinator (then bundle?) or is it possible to concurrently run both workflows
under one coordinator?
Thank you,
Dar
On 3/20/18, 2:25 PM, "Andras Piros" <and...@apache.org> wrote:
Hello,
does the RollbackException also occur when using a database other than
Derby, e.g. MySQL?
I'm also wondering why you use bash scripts instead of *Oozie coordinators
<https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html>*.
Regards,
Andras
On Mon, Mar 19, 2018 at 6:41 PM, Dar Swift <dar.sw...@dudesolutions.com>
wrote:
> Hello,
>
> I’ve been running two Oozie ssh-action workflows which are initiated via
> their own Java program. The two workflows have different names, but the
> steps in the workflows are identically named. The two workflows use
> different proxy users, the programs they run are in different directories
> and the data they access is different. Bash scripts that start each
> workflow are scheduled to start simultaneously on a Hadoop cluster using
> the “at” command. One job always runs successfully (not always the same
> job) and most of the time both jobs work. But often one job will fail.
>
> There are two different failure scenarios when a failure occurs.
> Sometimes Oozie will track the failed job attempt and the status will be
> stuck in PREP.
>
> Other times, Oozie logging doesn’t track the attempted job even though the
> Java program calling oozie returns “[ERROR] [main] StartOozie -
> java.lang.reflect.UndeclaredThrowableException: Unknown exception in
> doAs” the same error as jobs that get stuck in PREP.
>
> The following stack trace is from a job that was tracked by Oozie and
> stuck in PREP.
>
> Oozie is currently using Derby.
>
> Oozie Verion = 4.2.0
> Hadoop Version = 2.7.3
> Derby Version = 10.10.1.1
>
>
> Job Log:
>
> ERROR StartXCommand:517 - SERVER[XXXX] USER[XXXX] GROUP[-] TOKEN[]
> APP[ssh-wf-XXX] JOB[0000001-180318120929200-oozie-oozi-W] ACTION[]
> XException,
> org.apache.oozie.command.CommandException: E0603: SQL error in operation,
> <openjpa-2.2.2-r422266:1468616 fatal store error>
org.apache.openjpa.persistence.RollbackException:
> The transaction has been rolled back. See the nested exceptions for
> details on the errors that occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.oozie.command.wf.SignalXCommand.execute(
> SignalXCommand.java:448)
> at org.apache.oozie.command.wf.SignalXCommand.execute(
> SignalXCommand.java:82)
> at org.apache.oozie.command.XCommand.call(XCommand.java:
> 287)
> at org.apache.oozie.DagEngine.start(DagEngine.java:202)
> at org.apache.oozie.DagEngine.
> submitJob(DagEngine.java:116)
> at org.apache.oozie.servlet.V1JobsServlet.
> submitWorkflowJob(V1JobsServlet.java:192)
> at org.apache.oozie.servlet.V1JobsServlet.submitJob(
> V1JobsServlet.java:92)
> at org.apache.oozie.servlet.BaseJobsServlet.doPost(
> BaseJobsServlet.java:102)
> at javax.servlet.http.HttpServlet.service(
> HttpServlet.java:727)
> at org.apache.oozie.servlet.JsonRestServlet.service(
> JsonRestServlet.java:304)
> at javax.servlet.http.HttpServlet.service(
> HttpServlet.java:820)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(
> AuthFilter.java:171)
> at org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
> at org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
> at org.apache.oozie.servlet.AuthFilter.doFilter(
> AuthFilter.java:176)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.HostnameFilter.doFilter(
> HostnameFilter.java:86)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.OozieXFrameOptionsFilter.
> doFilter(OozieXFrameOptionsFilter.java:48)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.OozieCSRFFilter.doFilter(
> OozieCSRFFilter.java:62)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(
> StandardWrapperValve.java:234)
> at org.apache.catalina.core.StandardContextValve.invoke(
> StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(
> StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(
> ErrorReportValve.java:103)
> at org.apache.catalina.core.StandardEngineValve.invoke(
> StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(
> CoyoteAdapter.java:293)
> at org.apache.coyote.http11.Http11Processor.process(
> Http11Processor.java:859)
> at org.apache.coyote.http11.Http11Protocol$
> Http11ConnectionHandler.process(Http11Protocol.java:610)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(
> JIoEndpoint.java:503)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.oozie.executor.jpa.JPAExecutorException: E0603: SQL
> error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has
> been rolled back. See the nested exceptions for details on the errors
that
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.oozie.service.JPAService.
> executeBatchInsertUpdateDelete(JPAService.java:439)
> at org.apache.oozie.executor.jpa.BatchQueryExecutor.
> executeBatchInsertUpdateDelete(BatchQueryExecutor.java:132)
> at org.apache.oozie.command.wf.SignalXCommand.execute(
> SignalXCommand.java:439)
> ... 37 more
> Caused by: <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has
> been rolled back. See the nested exceptions for details on the errors
that
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.openjpa.persistence.EntityManagerImpl.
> commit(EntityManagerImpl.java:594)
> at org.apache.oozie.service.JPAService.
> executeBatchInsertUpdateDelete(JPAService.java:435)
> ... 39 more
> Caused by: <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.EntityNotFoundException: The transaction
> has been rolled back. See the nested exceptions for details on the errors
> that occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.openjpa.kernel.BrokerImpl.newFlushException(
> BrokerImpl.java:2347)
> at org.apache.openjpa.kernel.BrokerImpl.flush(BrokerImpl.
> java:2184)
> at org.apache.openjpa.kernel.BrokerImpl.flushSafe(
> BrokerImpl.java:2082)
> at org.apache.openjpa.kernel.BrokerImpl.beforeCompletion(
> BrokerImpl.java:2000)
> at org.apache.openjpa.kernel.LocalManagedRuntime.commit(
> LocalManagedRuntime.java:81)
> at org.apache.openjpa.kernel.BrokerImpl.commit(BrokerImpl.
> java:1524)
> at org.apache.openjpa.kernel.DelegatingBroker.commit(
> DelegatingBroker.java:933)
> at org.apache.openjpa.persistence.EntityManagerImpl.
> commit(EntityManagerImpl.java:570)
> ... 40 more
> Caused by: <openjpa-2.2.2-r422266:1468616 nonfatal store error>
> org.apache.openjpa.persistence.EntityNotFoundException: The instance of
> type "class org.apache.oozie.WorkflowActionBean" with oid
> "0000001-180318120929200-oozie-oozi-W@:start:" no longer exists in the
> data store. This may mean that you deleted the instance in a separate
> transaction, but this context still has a cached version.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.openjpa.kernel.StateManagerImpl.loadFields(
> StateManagerImpl.java:3109)
> at org.apache.openjpa.kernel.StateManagerImpl.loadField(
> StateManagerImpl.java:3185)
> at org.apache.openjpa.kernel.StateManagerImpl.
> fetchStringField(StateManagerImpl.java:2474)
> at org.apache.openjpa.kernel.StateManagerImpl.fetchString(
> StateManagerImpl.java:2464)
> at org.apache.openjpa.jdbc.meta.
> strats.StringFieldStrategy.insert(StringFieldStrategy.java:105)
> at org.apache.openjpa.jdbc.meta.FieldMapping.insert(
> FieldMapping.java:623)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> insert(AbstractUpdateManager.java:239)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> populateRowManager(AbstractUpdateManager.java:166)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> flush(AbstractUpdateManager.java:97)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> flush(AbstractUpdateManager.java:78)
> at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.flush(
> JDBCStoreManager.java:732)
> at org.apache.openjpa.kernel.DelegatingStoreManager.flush(
> DelegatingStoreManager.java:131)
> ... 47 more
> 2018-03-18 12:20:02,465 WARN V1JobsServlet:523 - SERVER[XXXX] USER[XXX]
> GROUP[-] TOKEN[] APP[ssh-wf-XXX] JOB[0000001-180318120929200-oozie-oozi-W]
> ACTION[] URL[POST http://XXX.XXX.XXX.XXX:11000/oozie/v2/jobs?action=start]
> error[E0603], E0603: SQL error in operation,
<openjpa-2.2.2-r422266:1468616
> fatal store error> org.apache.openjpa.persistence.RollbackException: The
> transaction has been rolled back. See the nested exceptions for details
on
> the errors that occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> org.apache.oozie.servlet.XServletException: E0603: SQL error in
> operation, <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has
> been rolled back. See the nested exceptions for details on the errors
that
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.oozie.servlet.V1JobsServlet.
> submitWorkflowJob(V1JobsServlet.java:197)
> at org.apache.oozie.servlet.V1JobsServlet.submitJob(
> V1JobsServlet.java:92)
> at org.apache.oozie.servlet.BaseJobsServlet.doPost(
> BaseJobsServlet.java:102)
> at javax.servlet.http.HttpServlet.service(
> HttpServlet.java:727)
> at org.apache.oozie.servlet.JsonRestServlet.service(
> JsonRestServlet.java:304)
> at javax.servlet.http.HttpServlet.service(
> HttpServlet.java:820)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(
> AuthFilter.java:171)
> at org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
> at org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
> at org.apache.oozie.servlet.AuthFilter.doFilter(
> AuthFilter.java:176)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.HostnameFilter.doFilter(
> HostnameFilter.java:86)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.OozieXFrameOptionsFilter.
> doFilter(OozieXFrameOptionsFilter.java:48)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.oozie.servlet.OozieCSRFFilter.doFilter(
> OozieCSRFFilter.java:62)
> at org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(ApplicationFilterChain.java:235)
> at org.apache.catalina.core.ApplicationFilterChain.
> doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(
> StandardWrapperValve.java:234)
> at org.apache.catalina.core.StandardContextValve.invoke(
> StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(
> StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(
> ErrorReportValve.java:103)
> at org.apache.catalina.core.StandardEngineValve.invoke(
> StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(
> CoyoteAdapter.java:293)
> at org.apache.coyote.http11.Http11Processor.process(
> Http11Processor.java:859)
> at org.apache.coyote.http11.Http11Protocol$
> Http11ConnectionHandler.process(Http11Protocol.java:610)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(
> JIoEndpoint.java:503)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.oozie.DagEngineException: E0603: SQL error in
> operation, <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has
> been rolled back. See the nested exceptions for details on the errors
that
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.oozie.DagEngine.start(DagEngine.java:205)
> at org.apache.oozie.DagEngine.
> submitJob(DagEngine.java:116)
> at org.apache.oozie.servlet.V1JobsServlet.
> submitWorkflowJob(V1JobsServlet.java:192)
> ... 32 more
> Caused by: org.apache.oozie.command.CommandException: E0603: SQL error in
> operation, <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has
> been rolled back. See the nested exceptions for details on the errors
that
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.oozie.command.wf.SignalXCommand.execute(
> SignalXCommand.java:448)
> at org.apache.oozie.command.wf.SignalXCommand.execute(
> SignalXCommand.java:82)
> at org.apache.oozie.command.XCommand.call(XCommand.java:
> 287)
> at org.apache.oozie.DagEngine.start(DagEngine.java:202)
> ... 34 more
> Caused by: org.apache.oozie.executor.jpa.JPAExecutorException: E0603: SQL
> error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has
> been rolled back. See the nested exceptions for details on the errors
that
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.oozie.service.JPAService.
> executeBatchInsertUpdateDelete(JPAService.java:439)
> at org.apache.oozie.executor.jpa.BatchQueryExecutor.
> executeBatchInsertUpdateDelete(BatchQueryExecutor.java:132)
> at org.apache.oozie.command.wf.SignalXCommand.execute(
> SignalXCommand.java:439)
> ... 37 more
> Caused by: <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has
> been rolled back. See the nested exceptions for details on the errors
that
> occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.openjpa.persistence.EntityManagerImpl.
> commit(EntityManagerImpl.java:594)
> at org.apache.oozie.service.JPAService.
> executeBatchInsertUpdateDelete(JPAService.java:435)
> ... 39 more
> Caused by: <openjpa-2.2.2-r422266:1468616 fatal store error>
> org.apache.openjpa.persistence.EntityNotFoundException: The transaction
> has been rolled back. See the nested exceptions for details on the errors
> that occurred.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.openjpa.kernel.BrokerImpl.newFlushException(
> BrokerImpl.java:2347)
> at org.apache.openjpa.kernel.BrokerImpl.flush(BrokerImpl.
> java:2184)
> at org.apache.openjpa.kernel.BrokerImpl.flushSafe(
> BrokerImpl.java:2082)
> at org.apache.openjpa.kernel.BrokerImpl.beforeCompletion(
> BrokerImpl.java:2000)
> at org.apache.openjpa.kernel.LocalManagedRuntime.commit(
> LocalManagedRuntime.java:81)
> at org.apache.openjpa.kernel.BrokerImpl.commit(BrokerImpl.
> java:1524)
> at org.apache.openjpa.kernel.DelegatingBroker.commit(
> DelegatingBroker.java:933)
> at org.apache.openjpa.persistence.EntityManagerImpl.
> commit(EntityManagerImpl.java:570)
> ... 40 more
> Caused by: <openjpa-2.2.2-r422266:1468616 nonfatal store error>
> org.apache.openjpa.persistence.EntityNotFoundException: The instance of
> type "class org.apache.oozie.WorkflowActionBean" with oid
> "0000001-180318120929200-oozie-oozi-W@:start:" no longer exists in the
> data store. This may mean that you deleted the instance in a separate
> transaction, but this context still has a cached version.
> FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.openjpa.kernel.StateManagerImpl.loadFields(
> StateManagerImpl.java:3109)
> at org.apache.openjpa.kernel.StateManagerImpl.loadField(
> StateManagerImpl.java:3185)
> at org.apache.openjpa.kernel.StateManagerImpl.
> fetchStringField(StateManagerImpl.java:2474)
> at org.apache.openjpa.kernel.StateManagerImpl.fetchString(
> StateManagerImpl.java:2464)
> at org.apache.openjpa.jdbc.meta.
> strats.StringFieldStrategy.insert(StringFieldStrategy.java:105)
> at org.apache.openjpa.jdbc.meta.FieldMapping.insert(
> FieldMapping.java:623)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> insert(AbstractUpdateManager.java:239)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> populateRowManager(AbstractUpdateManager.java:166)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> flush(AbstractUpdateManager.java:97)
> at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.
> flush(AbstractUpdateManager.java:78)
> at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.flush(
> JDBCStoreManager.java:732)
> at org.apache.openjpa.kernel.DelegatingStoreManager.flush(
> DelegatingStoreManager.java:131)
> ... 47 more
>
> Output Log:
>
> ERROR security.UserGroupInformation: PriviledgedActionException as:XXX via
> XXX cause:E0603 : E0603
> : SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store
> error> org.apache.openjpa.persistence.RollbackExcepti
> on: The transaction has been rolled back. See the nested exceptions for
> details on the errors that occurred. FailedObject
> : org.apache.oozie.WorkflowActionBean@66ff6081
> java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1203)
> at com.dsi.calloozie.StartOozie.main(StartOozie.java:54)
> Caused by: java.security.PrivilegedActionException: E0603 : E0603: SQL
> error in operation, <openjpa-2.2.2-r422266:1468616
> fatal store error> org.apache.openjpa.persistence.RollbackException: The
> transaction has been rolled back. See the nested
> exceptions for details on the errors that occurred. FailedObject:
> org.apache.oozie.WorkflowActionBean@66ff6081
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1190)
> ... 1 more
> Caused by: E0603 : E0603: SQL error in operation,
> <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.per
> sistence.RollbackException: The transaction has been rolled back. See the
> nested exceptions for details on the errors tha
> t occurred. FailedObject: org.apache.oozie.WorkflowActionBean@66ff6081
> at org.apache.oozie.client.OozieClient.handleError(
> OozieClient.java:542)
> at org.apache.oozie.client.OozieClient$JobSubmit.call(
> OozieClient.java:625)
> at org.apache.oozie.client.OozieClient$JobSubmit.call(
> OozieClient.java:595)
> at org.apache.oozie.client.OozieClient$ClientCallable.
> call(OozieClient.java:514)
> at org.apache.oozie.client.OozieClient.run(OozieClient.
> java:756)
> at com.dsi.calloozie.StartOozie$1.run(StartOozie.java:57)
> at com.dsi.calloozie.StartOozie$1.run(StartOozie.java:54)
> ... 4 more
>
>
