[
https://issues.apache.org/jira/browse/DERBY-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697833#action_12697833
]
Rick Hillegas commented on DERBY-4127:
--------------------------------------
I have looked into this problem and would like to summarize what I have
observed:
1) Simple test case
2) Scope of problem
3) Suspect code
4) Proposed solution (10.5)
5) Future improvements (10.6)
------- (1) Simple Test Case ---------------
Here is a simple way to reproduce this problem:
A) Hard-upgrade a 10.4 (or earlier) database to 10.5
B) Bring up the 10.5 network server
C) Run the following query through ij over the network:
select * from sys.sysroutineperms
I didn't spend any time looking into the ugly network error. I wrote it off as
just another example of the obscure diagnostics which the network layer barks
when it trips over a puzzling condition. I believe that the underlying problem
which muddled the network layer was the presence of a null in a non-nullable
column.
More specifically, the same query run through the embedded driver shows a
corrupt permission tuple. The PUBLIC permission granted to
SYSCS_UPDATE_STATISTICS has a null GRANTOR column. GRANTOR is a non-nullable
column and it should contain the name of the database owner.
The tuple should be
( permissionID = someUUID, grantee = PUBLIC, grantor = databaseOwner, aliasID =
updateStatisticsUUID, grantOption = N )
but is instead
( permissionID = someUUID, grantee = PUBLIC, grantor = NULL, aliasID =
updateStatisticsUUID, grantOption = N )
An extra complication of this corruption occurs if a database is hard-upgraded
from 10.0 to 10.5. In that case, the metadata holds two versions of the
permission: the correct version and the corrupt version.
------- (2) Scope of the Problem ---------------
I believe that this corrupt metadata is benign for the moment. That is, I don't
believe that this corruption causes Derby to malfunction in any serious
way--other than exposing some brittleness in the network layer. However, I
think that corrupt metadata is a ticking timebomb and we should not make the
metadata more corrupt than it already is.
In investigating this problem, I discovered another example of this corruption
which may have crept into Derby databases already. The permission granted to
SYSCS_INPLACE_COMPRESS_TABLE has a null GRANTOR column if the database was
hard-upgraded from 10.0 to 10.2 or later. Note that the corruption does not
appear in databases which underwent an intermediate upgrade to 10.1 before
being hard-upgraded to a later release. So the corrupt upgrade trajectories are:
10.0 -> 10.2
10.0 -> 10.3
10.0 -> 10.4
but the following trajectories are ok:
10.0 -> 10.1 -> 10.2
10.0 -> 10.1 -> 10.3
10.0 -> 10.1 -> 10.4
------- (3) Suspect Code ---------------
To recap, we see two upgrade problems:
i) A NULL GRANTOR column in a permission tuple--seen in all hard-upgrades to
10.5.
ii) Redundant permission tuples--seen when you hard-upgrade from 10.0.
I believe these problems are caused by the following issues in Derby's metadata
management:
i) We calculate the database owner AFTER upgrade. I think this was a deliberate
decision. Calculating the database owner AFTER upgrade gives you the correct
owner when you hard-upgrade to 10.2 or later. That is because 10.2 changes the
database owner. However, this late calculation means that the database owner is
NULL during upgrade--and that is why we stuff NULL into the GRANTOR column.
ii) We declare the list of PUBLIC routines twice: first in a tidy table and
then again in procedural logic. This redundant declaration results in multiple
copies of the same permission.
------- (4) Proposed solution (10.5) ---------------
I recommend the following immediate changes to 10.5 and the trunk:
i) Calculate the database owner BEFORE the database is upgraded. Then
recalculate the database owner during upgrade if we have to run the 10.2 logic
which changes the database owner.
ii) Declare the list of PUBLIC routines only once. I think this means removing
one of the existing declarations. Removing the tidy tables requires fewer code
changes but results in code that is harder to understand. Removing the
procedural redundancies causes a lot of edits but leaves you with a tidy
declaration of the PUBLIC routines which you understand at a glance. I vote for
this solution.
------- (5) Future improvements (10.6) ---------------
Even after making the changes in (4), we are still left with the corrupt
permission for SYSCS_INPLACE_COMPRESS_TABLE in databases which were
hard-upgraded as follows:
10.0 -> 10.2
10.0 -> 10.3
10.0 -> 10.4
We could add some 10.5 upgrade logic which corrects this corruption. However,
since I think the corruption is benign for the moment and no-one has complained
about it yet, I recommend that we defer this correction until the 10.6 upgrade.
I also recommend that we write a comprehensive test which compares virgin
metadata with hard-upgraded metadata. We should run this test on all
hard-upgrade trajectories. This may uncover other corruptions in Derby
metadata. We should correct those other corruptions in 10.6 too.
> ArrayIndexOutOfBoundsException in DatabaseMetaDataTest when run in Soft
> Upgrade mode
> ------------------------------------------------------------------------------------
>
> Key: DERBY-4127
> URL: https://issues.apache.org/jira/browse/DERBY-4127
> Project: Derby
> Issue Type: Bug
> Components: Regression Test Failure
> Affects Versions: 10.5.1.0
> Environment: Windows Vista 64, Junit 3.8.2, Sun JDK 1.6.0_10
> Reporter: Suran Jayathilaka
> Assignee: Rick Hillegas
> Priority: Blocker
> Attachments: Test4127.java
>
>
> This bug was found when doing soft upgrade testing from Derby version
> 10.4.2.0 to 10.5.1.0 (RC1)
> Steps followed are as follows.
> 1. Run setEmbeddedCP.bat from version 10.4.2.0's bin folder
> 2. In a test folder run ij
> 3. create system/wombat database.
> ij> connect 'jdbc:derby:system/wombat;create=true';
> 4. exit ij
> 5. Copy the 10.5.1.0 derby jars (from lib folder) and the derbyTesting.jar
> from 10.4.2.0 to the test folder and set classpath with them (including junit
> and ORO)
> 6. Run suites.All
> java -Xmx512M -Xms512M -Dderby.tests.trace=true junit.textui.TestRunner
> org.apache.derbyTesting.functionTests.suites.All
> Result:
> Tests run: 10479, Failures: 56, Errors: 34
> The stack trace is as follows.
> ---------------------------------------------------------------------------------------------------------
> 21)
> testGetColumnsReadOnly(org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest)java.lang.ArrayIndexOutOfBoundsException:
> 122
> at org.apache.derby.client.net.NetCursor.readFdocaBytes(Unknown Source)
> at org.apache.derby.client.net.NetCursor.parseSQLCAXGRP(Unknown Source)
> at org.apache.derby.client.net.NetCursor.parseSQLCAGRP(Unknown Source)
> at org.apache.derby.client.net.NetCursor.parseSQLCARD(Unknown Source)
> at
> org.apache.derby.client.net.NetCursor.calculateColumnOffsetsForRow_(Unknown
> Source)
> at org.apache.derby.client.am.Cursor.stepNext(Unknown Source)
> at
> org.apache.derby.client.net.NetCursor.scanDataBufferForEndOfData(Unknown
> Source)
> at org.apache.derby.client.net.NetResultSet.preClose_(Unknown Source)
> at org.apache.derby.client.am.ResultSet.closeX(Unknown Source)
> at org.apache.derby.client.am.ResultSet.close(Unknown Source)
> at
> org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.crossCheckGetColumnsAndResultSetMetaData(DatabaseMetaDataTest.java:1660)
> at
> org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.testGetColumnsReadOnly(DatabaseMetaDataTest.java:1357)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at
> org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:102)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at
> org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at
> org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> 22)
> testGetColumnsModify(org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest)java.lang.ArrayIndexOutOfBoundsException:
> 122
> at org.apache.derby.client.net.NetCursor.readFdocaBytes(Unknown Source)
> at org.apache.derby.client.net.NetCursor.parseSQLCAXGRP(Unknown Source)
> at org.apache.derby.client.net.NetCursor.parseSQLCAGRP(Unknown Source)
> at org.apache.derby.client.net.NetCursor.parseSQLCARD(Unknown Source)
> at
> org.apache.derby.client.net.NetCursor.calculateColumnOffsetsForRow_(Unknown
> Source)
> at org.apache.derby.client.am.Cursor.stepNext(Unknown Source)
> at
> org.apache.derby.client.net.NetCursor.scanDataBufferForEndOfData(Unknown
> Source)
> at org.apache.derby.client.net.NetResultSet.preClose_(Unknown Source)
> at org.apache.derby.client.am.ResultSet.closeX(Unknown Source)
> at org.apache.derby.client.am.ResultSet.close(Unknown Source)
> at
> org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.crossCheckGetColumnsAndResultSetMetaData(DatabaseMetaDataTest.java:1660)
> at
> org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.testGetColumnsReadOnly(DatabaseMetaDataTest.java:1357)
> at
> org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.testGetColumnsModify(DatabaseMetaDataTest.java:1374)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at
> org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:102)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at
> org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at
> org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> ----------------------------------------------------------------------------------
> I did not see anything in the derby.log from the tests that seemed to be
> related to this.
> Note that this test passed when run by itself, and also when run after a
> couple of failing tests (namely
> org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest and
> org.apache.derbyTesting.functionTests.tests.jdbcapi.UpdatableResultSetTest)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
