Aaron Baff created FLUME-1636:
---------------------------------
Summary: JDBC Channel receiving deadlock transaction rollback
Key: FLUME-1636
URL: https://issues.apache.org/jira/browse/FLUME-1636
Project: Flume
Issue Type: Bug
Components: Channel
Affects Versions: v1.1.0
Environment: Running on Linux with Java 1.6.0u21 (Oracle Hotspot JVM),
CDH 4.0.1 version of Flume v1.1.0.
Reporter: Aaron Baff
Seeing the following error which causes Avro clients to disconnect (I'm
guessing due to transaction failure), failover to another agent, then
eventually reconnect to the original agent.
#JDBC Channel Configuration
collector_agent.channels.defaultChannel.type = jdbc
collector_agent.channels.defaultChannel.db.type = DERBY
collector_agent.channels.defaultChannel.sysprop.user.home =
/f/flume-channels/default/
collector_agent.channels.defaultChannel.capacity = 0
collector_agent.channels.defaultChannel.maximum.capacity = 0
#Log File error
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for
rollback
2012-10-01 10:34:09,267 WARN impl.JdbcTransactionImpl: Marking transaction for
rollback
2012-10-01 10:34:09,267 INFO impl.JdbcTransactionImpl: Attempting transaction
roll-back
2012-10-01 10:34:09,268 ERROR source.AvroSource: Unable to process event batch
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:315)
at org.apache.flume.channel.jdbc.JdbcChannel.put(JdbcChannel.java:44)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:187)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:229)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:140)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:125)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to
persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@1b6ca9f0
at
org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:308)
... 30 more
Caused by: java.sql.SQLTransactionRollbackException: A lock could not be
obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT,
FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD,
FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown Source)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:646)
... 31 more
Caused by: java.sql.SQLException: A lock could not be obtained due to a
deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT,
FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD,
FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 44 more
Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle
of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,43)
Waiting XID : {242883779, X} , SA, INSERT INTO FLUME.FL_HEADER (FLH_EVENT,
FLH_NAME, FLH_VALUE, FLH_NMSPILL, FLH_VLSPILL) VALUES ( ?, ?, ?, ?, ?)
Granted XID : {242883747, X}
Lock : ROW, SYSCOLUMNS, (4,37)
Waiting XID : {242883747, X} , SA, INSERT INTO FLUME.FL_EVENT (FLE_PAYLOAD,
FLE_CHANNEL, FLE_SPILL) VALUES ( ?, ?, ?)
Granted XID : {242883779, X}
. The selected victim is XID : 242883779.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown
Source)
at
org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown
Source)
at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown
Source)
at
org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
at
org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(Unknown
Source)
at
org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.lockPositionForWrite(Unknown
Source)
at
org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(Unknown
Source)
at
org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSetAutoincrementValue(Unknown
Source)
at
org.apache.derby.impl.sql.execute.InsertResultSet.getSetAutoincrementValue(Unknown
Source)
at
org.apache.derby.impl.sql.execute.BaseActivation.getSetAutoincrementValue(Unknown
Source)
at
org.apache.derby.exe.ace8b858bfx013ax1d43x6465x000004f30e0821d.e0(Unknown
Source)
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown
Source)
at
org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
at
org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown
Source)
at
org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown
Source)
at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown
Source)
at
org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown
Source)
... 38 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
