[
https://issues.apache.org/jira/browse/DERBY-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490107
]
Kristian Waagan commented on DERBY-2566:
----------------------------------------
I would suggest someone with good knowledge of the object allocations down in
store/access took a look at the heap for this one.
One way to do this would be to use the Java SE 6 platform, run Java with
-XX:+HeapDumpOnOutOfMemoryError and then use jhat (remember to increase the
heap size for jhat with -J-XmxNNNm).
For a quick reference, here are the topmost results with TYPE_FORWARD_ONLY:
242175 instances of class [C
241126 instances of class [[C
241023 instances of class org.apache.derby.iapi.types.SQLVarchar
232557 instances of class java.lang.String
131700 instances of class org.apache.derby.impl.sql.GenericParameter
21353 instances of class org.apache.derby.impl.store.raw.data.StoredRecordHeader
18636 instances of class org.apache.derby.iapi.types.SQLLongint
18126 instances of class org.apache.derby.iapi.types.SQLTimestamp
17843 instances of class org.apache.derby.iapi.types.SQLDouble
10343 instances of class [Lorg.apache.derby.iapi.types.DataValueDescriptor;
9344 instances of class org.apache.derby.impl.sql.execute.ValueRow
9060 instances of class org.apache.derby.iapi.types.SQLRef
8866 instances of class [Ljava.util.HashMap$Entry;
8863 instances of class java.util.HashMap
8808 instances of class java.util.HashSet
8781 instances of class org.apache.derby.impl.sql.GenericParameterValueSet
8781 instances of class [Lorg.apache.derby.iapi.sql.execute.ExecRow;
8781 instances of class [Lorg.apache.derby.impl.sql.GenericParameter;
8780 instances of class
org.apache.derby.exe.ac07170079x0112x0ab4x5432x0000003d4b882
8780 instances of class org.apache.derby.impl.sql.execute.CurrentOfResultSet
7909 instances of class org.apache.derby.impl.store.raw.data.RecordId
4100 instances of class [B
For a quick reference, here are the topmost results with
TYPE_SCROLL_INSENSITIVE:
Instance Counts for All Classes (including platform)
310979 instances of class [[C
310876 instances of class org.apache.derby.iapi.types.SQLVarchar
174638 instances of class [C
162253 instances of class java.lang.String
84765 instances of class org.apache.derby.impl.sql.GenericParameter
34000 instances of class [Lorg.apache.derby.iapi.types.DataValueDescriptor;
33916 instances of class org.apache.derby.iapi.types.SQLTimestamp
22766 instances of class org.apache.derby.iapi.types.SQLLongint
22609 instances of class org.apache.derby.iapi.types.SQLDouble
17113 instances of class [Z
16958 instances of class org.apache.derby.impl.sql.execute.ValueRow
14337 instances of class org.apache.derby.impl.store.raw.data.StoredRecordHeader
11575 instances of class [B
11306 instances of class
org.apache.derby.impl.store.access.conglomerate.OpenConglomerateScratchSpace
11303 instances of class org.apache.derby.impl.sql.execute.IndexRow
11302 instances of class org.apache.derby.iapi.types.SQLRef
8693 instances of class org.apache.derby.impl.store.raw.data.RecordId
5786 instances of class org.apache.derby.iapi.services.io.FormatableBitSet
5744 instances of class [Ljava.util.HashMap$Entry;
5741 instances of class java.util.HashMap
If I change the repro to commit for every 500 updates, it runs without error
with SCROLL_INSENSITIVE. However, it performs a lot of GC (also some full GC),
but maybe this is to be expected in this case?
Last, it is not clear to me what is going on after the update phase. The repro
takes a long time to finish up after all the records have been updated;
rs.close - 16 s, stmt.close - 0 s, con.close - 31 s. CPU utilization is 100% in
this period. Traversing a long list of some kind?
Even with a commit for every 500 updates, the repro does not complete with
TYPE_FORWARD_ONLY. GC goes crazy...
Anyone got any ideas about this?
(FYI, I won't be working on this issue.)
> OutOfMemory/Sanity-assert failed when updating database
> -------------------------------------------------------
>
> Key: DERBY-2566
> URL: https://issues.apache.org/jira/browse/DERBY-2566
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.1.3.1, 10.2.2.0, 10.3.0.0
> Environment: Sun JDK 1.6-b105, Linux 2.6.20
> Sun JDK 1.5, Solaris 10
> Reporter: Kurt Huwig
> Attachments: Derby2566Repro.java
>
>
> If you run this application:
> -------- 8< -------
> import java.sql.*;
> import java.util.*;
> public class DerbyBug4 {
> private static String x;
> private static Random random = new Random(0);
> private static final int DB_SIZE = 10000;
> public static void main(String[] args) throws Exception {
> final StringBuilder sbX = new StringBuilder();
> for (int i = 0; i < 65535; i++) {
> sbX.append('X');
> }
> x = sbX.toString();
> Class.forName("org.apache.derby.jdbc.EmbeddedDriver");
> final Connection con =
> DriverManager.getConnection("jdbc:derby:bug;create=true");
> createAndFillDatabase(con);
> updateDatabase(con);
> con.close();
> }
> private static void updateDatabase(final Connection con) throws
> SQLException {
> final Statement stm = con.createStatement(ResultSet.TYPE_FORWARD_ONLY,
> ResultSet.CONCUR_UPDATABLE);
> final ResultSet rs = stm.executeQuery("SELECT * FROM journal FOR
> UPDATE");
> for (int row = 1; rs.next(); row++) {
> update(rs, "ip");
> update(rs, "sender");
> update(rs, "recipient");
> update(rs, "mailsender");
> update(rs, "mailfrom");
> update(rs, "mailto");
> update(rs, "cc");
> update(rs, "bcc");
> update(rs, "replyto");
> update(rs, "subject");
> update(rs, "attachments");
> update(rs, "status");
> update(rs, "reason");
> rs.updateInt("totallength", 0);
> rs.updateDouble("spamscore", 0);
> rs.updateRow();
> if (row % 500 == 0) {
> System.out.println("update: " + row);
> }
> }
> rs.close();
> stm.close();
> }
> private static void createAndFillDatabase(Connection con)
> throws SQLException {
> final Statement stm = con.createStatement();
> stm.executeUpdate("CREATE TABLE journal("
> + "ID VARCHAR(20) PRIMARY KEY default '' NOT NULL,"
> + "IP VARCHAR(45) default '' NOT NULL,"
> + "SENDER VARCHAR(32000) default '' NOT NULL,"
> + "RECIPIENT VARCHAR(32000) default '' NOT NULL,"
> + "MAILSENDER VARCHAR(32000) default '' NOT NULL,"
> + "MAILFROM VARCHAR(32000) default '' NOT NULL,"
> + "MAILTO VARCHAR(32000) default '' NOT NULL,"
> + "CC VARCHAR(32000) default '' NOT NULL,"
> + "BCC VARCHAR(32000) default '' NOT NULL,"
> + "REPLYTO VARCHAR(32000) default '' NOT NULL,"
> + "MAILDATE TIMESTAMP default '0001-01-01 00:00:00',"
> + "RECEIVEDDATE TIMESTAMP default '0001-01-01 00:00:00' NOT
> NULL,"
> + "SUBJECT VARCHAR(32000) default '' NOT NULL,"
> + "TOTALLENGTH BIGINT default 0 NOT NULL,"
> + "ATTACHMENTS VARCHAR(32000) default '' NOT NULL,"
> + "SPAMSCORE DOUBLE NOT NULL,"
> + "STATUS VARCHAR(11) default 'aborted' NOT NULL,"
> + "REASON VARCHAR(32000) NOT NULL)");
> con.commit();
> stm.close();
>
> final PreparedStatement pstm = con.prepareStatement(
> "INSERT INTO journal
> VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)");
> for (int i = 0; i < DB_SIZE; i++) {
> int col = 1;
> pstm.setString(col++, Integer.toHexString(i));
> doRandom(pstm, col++, 45);
> doRandom(pstm, col++, 10);
> doRandom(pstm, col++, 10);
> doRandom(pstm, col++, 5);
> doRandom(pstm, col++, 20);
> doRandom(pstm, col++, 20);
> doRandom(pstm, col++, 40);
> doRandom(pstm, col++, 5);
> doRandom(pstm, col++, 20);
> pstm.setTimestamp(col++, random.nextBoolean()
> ? new
> Timestamp(System.currentTimeMillis())
> : null);
> pstm.setTimestamp(col++, new
> Timestamp(System.currentTimeMillis()));
> doRandom(pstm, col++, 80);
> pstm.setInt(col++, random.nextInt(10000000));
> doRandom(pstm, col++, 500);
> pstm.setDouble(col++, random.nextDouble() * 10);
> doRandom(pstm, col++, 11);
> doRandom(pstm, col++, 100);
>
> pstm.executeUpdate();
> if (i % 500 == 0) {
> con.commit();
> System.out.println("insert: " + i);
> }
> }
> con.commit();
> pstm.close();
> }
> private static void doRandom(PreparedStatement pstm, int pos, int length)
> throws SQLException {
> pstm.setString(pos, x.substring(0, random.nextInt(length)));
> }
>
> private static void update(final ResultSet rs, final String field)
> throws SQLException {
> rs.updateString(field, x.substring(0, rs.getString(field).length()));
> }
> }
> -------- 8< -------
> you will get this exception:
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.services.io.DynamicByteArrayOutputStream.<init>(Unknown
> Source)
> at
> org.apache.derby.impl.store.raw.data.StoredPage.storeRecordForUpdate(Unknown
> Source)
> at org.apache.derby.impl.store.raw.data.StoredPage.storeRecord(Unknown
> Source)
> at org.apache.derby.impl.store.raw.data.UpdateOperation.doMe(Unknown
> Source)
> at org.apache.derby.impl.store.raw.log.FileLogger.logAndDo(Unknown
> Source)
> at org.apache.derby.impl.store.raw.xact.Xact.logAndDo(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.LoggableActions.doAction(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.LoggableActions.actionUpdate(Unknown
> Source)
> at
> org.apache.derby.impl.store.raw.data.StoredPage.doUpdateAtSlot(Unknown Source)
> at org.apache.derby.impl.store.raw.data.BasePage.updateAtSlot(Unknown
> Source)
> at
> org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.replace(Unknown
> Source)
> at org.apache.derby.impl.sql.execute.RowChangerImpl.updateRow(Unknown
> Source)
> at
> org.apache.derby.impl.sql.execute.UpdateResultSet.collectAffectedRows(Unknown
> Source)
> at org.apache.derby.impl.sql.execute.UpdateResultSet.open(Unknown
> Source)
> at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedResultSet.updateRow(Unknown Source)
> at DerbyBug4.updateDatabase(DerbyBug4.java:43)
> at DerbyBug4.main(DerbyBug4.java:19)
> If you enable sanity checks, then this exception occurs:
> Exception in thread "main"
> org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED
> statementContext is not expected to equal statementContexts[0]
> at
> org.apache.derby.shared.common.sanity.SanityManager.ASSERT(SanityManager.java:120)
> at
> org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.popStatementContext(GenericLanguageConnectionContext.java:2095)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.updateRow(EmbedResultSet.java:3773)
> at DerbyBug4.updateDatabase(DerbyBug4.java:43)
> at DerbyBug4.main(DerbyBug4.java:19)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
