Hi all, I recently rebased oozie to trunk and used mysql as the db. Upon submission of a job, I see this error on the command line: Error: E0803 : E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. The log in oozie.log shows: org.apache.oozie.command.CommandException: E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:252) at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:72) at org.apache.oozie.command.XCommand.call(XCommand.java:280) at org.apache.oozie.DagEngine.submitJob(DagEngine.java:109) at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:188) at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:88) at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:99) at javax.servlet.http.HttpServlet.service(HttpServlet.java:643) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:286) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:680) Caused by: org.apache.oozie.command.CommandException: E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:231) ... 29 more Caused by: org.apache.oozie.executor.jpa.JPAExecutorException: E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.service.JPAService.executeBatchInsertUpdateDelete(JPAService.java:392) at org.apache.oozie.executor.jpa.BatchQueryExecutor.executeBatchInsertUpdateDelete(BatchQueryExecutor.java:140) at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:228) ... 29 more Caused by: <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:594) at org.apache.oozie.service.JPAService.executeBatchInsertUpdateDelete(JPAService.java:388) ... 31 more Caused by: <openjpa-2.2.2-r422266:1468616 fatal general error> org.apache.openjpa.persistence.PersistenceException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.openjpa.kernel.BrokerImpl.newFlushException(BrokerImpl.java:2347) at org.apache.openjpa.kernel.BrokerImpl.flush(BrokerImpl.java:2184) at org.apache.openjpa.kernel.BrokerImpl.flushSafe(BrokerImpl.java:2082) at org.apache.openjpa.kernel.BrokerImpl.beforeCompletion(BrokerImpl.java:2000) at org.apache.openjpa.kernel.LocalManagedRuntime.commit(LocalManagedRuntime.java:81) at org.apache.openjpa.kernel.BrokerImpl.commit(BrokerImpl.java:1524) at org.apache.openjpa.kernel.DelegatingBroker.commit(DelegatingBroker.java:933) at org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:570) ... 32 more Caused by: <openjpa-2.2.2-r422266:1468616 nonfatal general error> org.apache.openjpa.persistence.PersistenceException: org.apache.oozie.BinaryBlob at org.apache.openjpa.util.Serialization.serialize(Serialization.java:67) at org.apache.openjpa.jdbc.sql.DBDictionary.serialize(DBDictionary.java:1510) at org.apache.openjpa.jdbc.sql.DBDictionary.setBlobObject(DBDictionary.java:1010) at org.apache.openjpa.jdbc.sql.DBDictionary.setTyped(DBDictionary.java:1337) at org.apache.openjpa.jdbc.sql.RowImpl.flush(RowImpl.java:896) at org.apache.openjpa.jdbc.sql.RowImpl.flush(RowImpl.java:856) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flushAndUpdate(PreparedStatementManagerImpl.java:117) at org.apache.openjpa.jdbc.kernel.BatchingPreparedStatementManagerImpl.flushAndUpdate(BatchingPreparedStatementManagerImpl.java:79) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flushInternal(PreparedStatementManagerImpl.java:100) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flush(PreparedStatementManagerImpl.java:88) at org.apache.openjpa.jdbc.kernel.ConstraintUpdateManager.flush(ConstraintUpdateManager.java:550) at org.apache.openjpa.jdbc.kernel.ConstraintUpdateManager.flush(ConstraintUpdateManager.java:106) at org.apache.openjpa.jdbc.kernel.BatchingConstraintUpdateManager.flush(BatchingConstraintUpdateManager.java:59) at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.flush(AbstractUpdateManager.java:105) at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.flush(AbstractUpdateManager.java:78) at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.flush(JDBCStoreManager.java:732) at org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:131) ... 39 more Caused by: <openjpa-2.2.2-r422266:1468616 nonfatal general error> org.apache.openjpa.persistence.PersistenceException: org.apache.oozie.BinaryBlob at org.apache.openjpa.util.Serialization.serialize(Serialization.java:67) at org.apache.openjpa.jdbc.sql.DBDictionary.serialize(DBDictionary.java:1510) at org.apache.openjpa.jdbc.sql.DBDictionary.setBlobObject(DBDictionary.java:1010) at org.apache.openjpa.jdbc.sql.DBDictionary.setTyped(DBDictionary.java:1337) at org.apache.openjpa.jdbc.sql.RowImpl.flush(RowImpl.java:896) at org.apache.openjpa.jdbc.sql.RowImpl.flush(RowImpl.java:856) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flushAndUpdate(PreparedStatementManagerImpl.java:117) at org.apache.openjpa.jdbc.kernel.BatchingPreparedStatementManagerImpl.flushAndUpdate(BatchingPreparedStatementManagerImpl.java:79) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flushInternal(PreparedStatementManagerImpl.java:100) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flush(PreparedStatementManagerImpl.java:88) at org.apache.openjpa.jdbc.kernel.ConstraintUpdateManager.flush(ConstraintUpdateManager.java:550) at org.apache.openjpa.jdbc.kernel.ConstraintUpdateManager.flush(ConstraintUpdateManager.java:106) at org.apache.openjpa.jdbc.kernel.BatchingConstraintUpdateManager.flush(BatchingConstraintUpdateManager.java:59) at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.flush(AbstractUpdateManager.java:105) at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.flush(AbstractUpdateManager.java:78) at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.flush(JDBCStoreManager.java:732) at org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:131) ... 39 more Caused by: java.io.NotSerializableException: org.apache.oozie.BinaryBlob at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at org.apache.openjpa.util.Serialization.serialize(Serialization.java:63) ... 55 more 2013-11-04 17:12:12,943 WARN V1JobsServlet:542 - SERVER[BZhangs-MacBook-Pro.local] USER[bzhang] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-131104170850462-oozie-bzha-W] ACTION[-] URL[POST http://localhost:11000/oozie/v2/jobs?action=start] error[E0803], E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apa che.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. org.apache.oozie.servlet.XServletException: E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:193) at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:88) at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:99) at javax.servlet.http.HttpServlet.service(HttpServlet.java:643) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:286) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:372) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:680) Caused by: org.apache.oozie.DagEngineException: E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.DagEngine.submitJob(DagEngine.java:116) at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:188) ... 25 more Caused by: org.apache.oozie.command.CommandException: E0803: IO error, E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:252) at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:72) at org.apache.oozie.command.XCommand.call(XCommand.java:280) at org.apache.oozie.DagEngine.submitJob(DagEngine.java:109) ... 26 more Caused by: org.apache.oozie.command.CommandException: E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:231) ... 29 more Caused by: org.apache.oozie.executor.jpa.JPAExecutorException: E0603: SQL error in operation, <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.oozie.service.JPAService.executeBatchInsertUpdateDelete(JPAService.java:392) at org.apache.oozie.executor.jpa.BatchQueryExecutor.executeBatchInsertUpdateDelete(BatchQueryExecutor.java:140) at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:228) ... 29 more Caused by: <openjpa-2.2.2-r422266:1468616 fatal store error> org.apache.openjpa.persistence.RollbackException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:594) at org.apache.oozie.service.JPAService.executeBatchInsertUpdateDelete(JPAService.java:388) ... 31 more Caused by: <openjpa-2.2.2-r422266:1468616 fatal general error> org.apache.openjpa.persistence.PersistenceException: The transaction has been rolled back. See the nested exceptions for details on the errors that occurred. at org.apache.openjpa.kernel.BrokerImpl.newFlushException(BrokerImpl.java:2347) at org.apache.openjpa.kernel.BrokerImpl.flush(BrokerImpl.java:2184) at org.apache.openjpa.kernel.BrokerImpl.flushSafe(BrokerImpl.java:2082) at org.apache.openjpa.kernel.BrokerImpl.beforeCompletion(BrokerImpl.java:2000) at org.apache.openjpa.kernel.LocalManagedRuntime.commit(LocalManagedRuntime.java:81) at org.apache.openjpa.kernel.BrokerImpl.commit(BrokerImpl.java:1524) at org.apache.openjpa.kernel.DelegatingBroker.commit(DelegatingBroker.java:933) at org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:570) ... 32 more Caused by: <openjpa-2.2.2-r422266:1468616 nonfatal general error> org.apache.openjpa.persistence.PersistenceException: org.apache.oozie.BinaryBlob at org.apache.openjpa.util.Serialization.serialize(Serialization.java:67) at org.apache.openjpa.jdbc.sql.DBDictionary.serialize(DBDictionary.java:1510) at org.apache.openjpa.jdbc.sql.DBDictionary.setBlobObject(DBDictionary.java:1010) at org.apache.openjpa.jdbc.sql.DBDictionary.setTyped(DBDictionary.java:1337) at org.apache.openjpa.jdbc.sql.RowImpl.flush(RowImpl.java:896) at org.apache.openjpa.jdbc.sql.RowImpl.flush(RowImpl.java:856) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flushAndUpdate(PreparedStatementManagerImpl.java:117) at org.apache.openjpa.jdbc.kernel.BatchingPreparedStatementManagerImpl.flushAndUpdate(BatchingPreparedStatementManagerImpl.java:79) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flushInternal(PreparedStatementManagerImpl.java:100) at org.apache.openjpa.jdbc.kernel.PreparedStatementManagerImpl.flush(PreparedStatementManagerImpl.java:88) at org.apache.openjpa.jdbc.kernel.ConstraintUpdateManager.flush(ConstraintUpdateManager.java:550) at org.apache.openjpa.jdbc.kernel.ConstraintUpdateManager.flush(ConstraintUpdateManager.java:106) at org.apache.openjpa.jdbc.kernel.BatchingConstraintUpdateManager.flush(BatchingConstraintUpdateManager.java:59) at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.flush(AbstractUpdateManager.java:105) at org.apache.openjpa.jdbc.kernel.AbstractUpdateManager.flush(AbstractUpdateManager.java:78) at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.flush(JDBCStoreManager.java:732) at org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:131) ... 39 more Caused by: java.io.NotSerializableException: org.apache.oozie.BinaryBlob at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at org.apache.openjpa.util.Serialization.serialize(Serialization.java:63) ... 55 more
Could this be related to oozie-1462? I don't see the config in oozie-site.xml to turn the compression off. Virag, can you take a look at it? I am using mysql 5.6. Bowen
