Re: Guidance/Help/Book/References?
Hi Pavel, Unfortunately, I'm not writing with any answers. My company's software also makes use of Derby in a high-concurrency setting, and we have encountered similar issues. For us, issues around dead-locks when simply preparing statements has been a major pain. We have indeed had to put in code-level synchronization in various places to try to deal with the issues. Unfortunately, I think you've run into bug 4279... https://issues.apache.org/jira/browse/DERBY-4279 This deadlock is unique because it can still occur in a properly designed database. You are only safe if all of your transactions are very simple and cannot be interleaved in a sequence that causes the deadlock, or if your particular statements do not require a table lock to compile. (For the sake of simplicity, I used LOCK TABLE in my example, but any UPDATE statement would fit.) I had proposed a patch to 4279, but ultimately had to abandon it due to complex and non-obvious synchronization issues. Maybe I'll take another cut at it... -Brett On Sat, Jun 23, 2012 at 1:18 AM, Pavel Bortnovskiy pbortnovs...@jefferies.com wrote: Hello, all: Derby is used heavily in my project and its tables are frequently accessed concurrently by multiple threads. Some threads update one or several tables, while other threads perform run select statements against those. I’ve written to this group several times whenever errors occurred, but some of those message have either been ignored or contained references to fairly short explanations on the website. So, the problems continue and it is difficult to find the proper solution. All problems happen due to the concurrent access to tables. For instance, today’s problem had to do with one thread updating one table (the table is first truncated and then re-populated to stay in sync with the source of data), while the other thread was trying to create a PreparedStatement. So, while one thread was truncating a table, another threw this exception (probably caused by nested select statements): java.sql.SQLException: The conglomerate (1,280) requested does not exist. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.init(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.init(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.init(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.init(Unknown Source) at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at com.jefco.fi.commons.database.SqlSyntaxStatement.init(SqlSyntaxStatement.java:76) at com.jefco.fi.commons.database.SqlSyntaxStatement.init(SqlSyntaxStatement.java:44) ... Caused by: java.sql.SQLException: The conglomerate (1,280) requested does not exist. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 19 more Caused by: ERROR XSAI2: The conglomerate (1,280) requested does not exist. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.access.heap.HeapConglomerateFactory.readConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.getStaticCompiledConglomInfo(Unknown Source) at org.apache.derby.impl.sql.compile.BaseJoinStrategy.fillInScanArgs1(Unknown Source) at org.apache.derby.impl.sql.compile.HashJoinStrategy.getScanArgs(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.getScanArguments(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.generateResultSet(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.generate(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(Unknown Source) at
Re: Guidance/Help/Book/References?
Derby is used heavily in my project and its tables are frequently accessed concurrently by multiple threads. Some threads update one or several tables, while other threads perform run select statements against those. I’ve written to this group several times whenever errors occurred, but some of those message have either been ignored or contained references to fairly short explanations on the website. So, the problems continue and it is difficult to find the proper solution. ... I must resolve all these errors ASAP and I would like to ask some guidance as to how to do it most properly. Avoiding deadlocks can be very challenging. I have often found that a very crude mechanism can be used. In my application, I modified my code to use Derby's LOCK TABLE statement. For each transaction, I identified which table(s) it was accessing. At the start of the transaction, my code issued a series of LOCK TABLE statements: LOCK TABLE t1 IN EXCLUSIVE MODE; LOCK TABLE t2 IN EXCLUSIVE MODE; I did this even for read-only transactions (and used EXCLUSIVE mode for them, although I certainly could have used SHARE mode). I thus single-threaded all my Derby accesses, and avoided all deadlocks. I encapsulated this logic into a common subroutine to avoid cluttering up the rest of the code; I just have to pass that subroutine the correct set of tables for this particular transaction, and the JDBC connection to use. I ensured that I always locked the tables in the same order (to avoid deadlocks). Although this meant that my transaction was somewhat less concurrent than it might have been: a) I never had to deal with unexpected deadlocks in the middle of a transaction b) I found that performance was just fine for my application If deadlocks are an ongoing persistent problem, you might consider this technique. thanks, bryan
Re: Guidance/Help/Book/References?
On 22/06/2012 19:18, Pavel Bortnovskiy wrote: Hello, all: Derby is used heavily in my project and its tables are frequently accessed concurrently by multiple threads. Some threads update one or several tables, while other threads perform run select statements against those. I’ve written to this group several times whenever errors occurred, but some of those message have either been ignored or contained references to fairly short explanations on the website. So, the problems continue and it is difficult to find the proper solution. All problems happen due to the concurrent access to tables. For instance, today’s problem had to do with one thread updating one table (the table is first truncated and then re-populated to stay in sync with the source of data), while the other thread was trying to create a PreparedStatement. So, while one thread was truncating a table, another threw this exception (probably caused by nested select statements): Hi Pavel, What transaction isolation level are you using for the connections involved? -- John English
Guidance/Help/Book/References?
Hello, all: Derby is used heavily in my project and its tables are frequently accessed concurrently by multiple threads. Some threads update one or several tables, while other threads perform run select statements against those. I've written to this group several times whenever errors occurred, but some of those message have either been ignored or contained references to fairly short explanations on the website. So, the problems continue and it is difficult to find the proper solution. All problems happen due to the concurrent access to tables. For instance, today's problem had to do with one thread updating one table (the table is first truncated and then re-populated to stay in sync with the source of data), while the other thread was trying to create a PreparedStatement. So, while one thread was truncating a table, another threw this exception (probably caused by nested select statements): java.sql.SQLException: The conglomerate (1,280) requested does not exist. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.init(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.init(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.init(Unknown Source) at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.init(Unknown Source) at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) at com.jefco.fi.commons.database.SqlSyntaxStatement.init(SqlSyntaxStatement.java:76) at com.jefco.fi.commons.database.SqlSyntaxStatement.init(SqlSyntaxStatement.java:44) ... Caused by: java.sql.SQLException: The conglomerate (1,280) requested does not exist. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 19 more Caused by: ERROR XSAI2: The conglomerate (1,280) requested does not exist. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.access.heap.HeapConglomerateFactory.readConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.getStaticCompiledConglomInfo(Unknown Source) at org.apache.derby.impl.sql.compile.BaseJoinStrategy.fillInScanArgs1(Unknown Source) at org.apache.derby.impl.sql.compile.HashJoinStrategy.getScanArgs(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.getScanArguments(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.generateResultSet(Unknown Source) at org.apache.derby.impl.sql.compile.FromBaseTable.generate(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(Unknown Source) at org.apache.derby.impl.sql.compile.GroupByNode.generate(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(Unknown Source) at org.apache.derby.impl.sql.compile.SubqueryNode.generateExpression(Unknown Source) at org.apache.derby.impl.sql.compile.BinaryOperatorNode.generateExpression(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(Unknown Source) at org.apache.derby.impl.sql.compile.UnionNode.generate(Unknown Source) at org.apache.derby.impl.sql.compile.NormalizeResultSetNode.generate(Unknown Source) at org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(Unknown Source)
