Daniel John Debrunner wrote:
Olav Sandstaa wrote:
I think I have found what causes the Nist tests to fail when running
derbyall with jdk16. The Nist tests started to fail after DERBY-930
(Add support for autoloading of Derby client drivers) was check in.
[snipped - good investigation work]
When the Class.forName("com.ibm.db2.jcc.DB2Driver") is executed (and
the DB2 driver is actually in the class path) it seems to call some
methods that happens to be defined by any JDBC driver (see also the
comment in the code). The class loader seems to touch derby.jar to try
to find these methods, and this makes the embedded driver to
"automagically" be loaded. And loading the embedded driver also
"automagically" attempts to boot an embedded Derby. Unfortunately (or
fortunately?) at this time the Derby properties has not been set for
defining an embedded Derby and the booting of Derby fails (silently).
I'm a little lost as to what you are describing here. What do you mean
when you say "attempts to boot an embedded Derby"? Loading the embedded
driver will always loaded the emebdded derby engine, unless it's already
booted. That's the function of the driver.
Then why does the driver boot fail? No properties are required to boot
the derby engine so maybe the auto-loading is broken in some way?
It seems like I was a bit lost here too :-( . After having looked a bit
more into the code for booting the engine it seems like I followed a
null pointer and ended up with a wrong conclusion. In the FileMonitor
class there is a variable named home of type File. When the Nist tests
runs correctly the home variable contains the directory for where the
tests should be running. In the case where the Nist tests fails the
FileMonitor's home is a null pointer, and since this caused the loading
of the StorageFactoryService to end up with a security exception I
wrongly assumed the cause for the security exception being this null
pointer. After having studied the code more carefully it is a legal
situation for this variable to be null. All it means is that the current
directory will be used for creating database in - I wrongly saw this as
an indication of the engine not properly being started.
About an hour or two later when the first Nist test starts it also
requests the embedded driver to be loaded into the same VM. In the
code for loading the driver and booting the database the following
check is done (see JDBCBoot.boot()):
if (org.apache.derby.jdbc.InternalDriver.activeDriver() == null)
{
// request that the InternalDriver (JDBC) service and the
// authentication service be started.
//
addProperty("derby.service.jdbc",
"org.apache.derby.jdbc.InternalDriver");
addProperty("derby.service.authentication",
AuthenticationService.MODULE);
Monitor.startMonitor(bootProperties, logging);
The test for checking if the driver has been loaded returns a pointer
to the embedded driver loaded during starting of derbyall. Based on
this no Derby database is created (since it based on having the
embedded driver expect that it also must have an embedded
database). So the Nist tests are run with the embedded driver but no
embedded database. No surprise that this make the tests fail
eventually (with an security exception seen by the test).
Loading the driver is separate from creating databases so again I'm
confused as to what you are describing here. Does the nist suite or some
part of the test harness set some properties that are only used at boot
time to create a database? Which properties are these?
The property that is causing the difference in behavior depending on
when the embedded driver is loaded is derby.system.home. There is no
error in the driver or engine with the regards to this, it is just that
with autoloading of the driver the value of the derby.system.home on the
time when the driver and engine is loaded influence on whether the Nist
tests succeeds or fails.
When the Nist tests succeeds, derby.system.home contains the name of the
directory where the test should be run (e.g.
/export/home/tmp/derbysuite/nist in my case). I think derby.system.home
is set by the test harness when it starts each individual test suite.
When the Nist tests fails the embedded driver and the Derby engine is
loaded as part of startup of derbyall (actually as part of loading the
DB2 driver). At this time the derby.system.home property is not set. It
think this leads to the following happening during autoloading of the
embedded Driver:
* The FileMonitor's home variable gets the value null based on
reading derby.system.home.
When the Nist tests start an hour later and they requests to get a
database created, the initialization of the StorageFactory fails with
the security exception since it uses the FileMonitor's home variable
(which is null) as the directory of where to create the database.
Without the security manager this would have succeeded, but with the
security manager, a security exception is raised when trying to get the
canonical path from the file system when only having the name of the
database directory (i.e. just having "wombat" instead of
"/export/....../wombat").
The exception raised is (see also earlier email in this thread):
java.security.AccessControlException: access denied
(java.util.PropertyPermission user.dir read)
at
java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
at
java.security.AccessController.checkPermission(AccessController.java:546)
at
java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at
java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1285)
at java.lang.System.getProperty(System.java:652)
at java.io.UnixFileSystem.resolve(UnixFileSystem.java:118)
at java.io.File.getCanonicalPath(File.java:559)
at
org.apache.derby.impl.io.DirStorageFactory.doInit(DirStorageFactory.java:194)
at
org.apache.derby.impl.io.BaseStorageFactory.init(BaseStorageFactory.java:90)
at
org.apache.derby.impl.services.monitor.StorageFactoryService.privGetStorageFactoryInstance(StorageFactoryService.java:240)
at
org.apache.derby.impl.services.monitor.StorageFactoryService.access$400(StorageFactoryService.java:64)
at
org.apache.derby.impl.services.monitor.StorageFactoryService$11.run(StorageFactoryService.java:813)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.derby.impl.services.monitor.StorageFactoryService.getCanonicalServiceName(StorageFactoryService.java:807)
at
org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1572)
at
org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:1025)
at
org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:543)
at
org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:1591)
at
org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:216)
at
org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:72)
at
org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:47)
at
org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:64)
at
org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:215)
at java.sql.DriverManager.getConnection(DriverManager.java:548)
at java.sql.DriverManager.getConnection(DriverManager.java:148)
at org.apache.derby.impl.tools.ij.util.startJBMS(util.java:518)
at org.apache.derby.impl.tools.ij.util.startJBMS(util.java:618)
at
org.apache.derby.impl.tools.ij.ConnectionEnv.init(ConnectionEnv.java:64)
at org.apache.derby.impl.tools.ij.utilMain.<init>(utilMain.java:176)
at
org.apache.derby.impl.tools.ij.utilMain14.<init>(utilMain14.java:51)
at
org.apache.derby.impl.tools.ij.Main14.getutilMain(Main14.java:102)
at org.apache.derby.impl.tools.ij.Main.<init>(Main.java:265)
at org.apache.derby.impl.tools.ij.Main14.<init>(Main14.java:68)
at org.apache.derby.impl.tools.ij.Main14.getMain(Main14.java:91)
at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:189)
at org.apache.derby.impl.tools.ij.Main14.main(Main14.java:55)
at org.apache.derby.tools.ij.main(ij.java:60)
at java.lang.Thread.run(Thread.java:619)
So basically the Nist tests fail due to the embedded driver is loaded
before derby.system.home is set to the value that it should have for the
test. And this leads to the security exception when the VM trying to
get the canonical path name for the current directory.
Seems like we are close to solving this with your work, but maybe
missing some information still.
I hope this made a more correct explanation of why the Nist tests fails.
So how should we fix this? Any suggestions? Some ideas:
* Remove the support for autoloading of the embedded driver (Yes, Rick
and the JDBC4 expert group have already said no to this, but I still
think there are situations where automagic loading of the drivers is a
bad idea)
* Change the test harness so that derby.system.home is set to a value
that is OK for all tests suites (running with useprocess=false) early
during initialization (before touching any JDBC driver code)
* Change the RunSuite(?) code so that it unloads the embedded driver
and the engine as part of the start up (to get rid of the autoloaded
embedded driver)
* Change the "security manager" properties for the Nist tests to avoid
the security exception when the VM tries to get the canonical path for
the working directory.
Any preferences or suggestions for better ideas?
Thanks,
Olav
