[jira] [Commented] (DERBY-7006) Investigate putting generated classes under the engine module loader
[
https://issues.apache.org/jira/browse/DERBY-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649095#comment-16649095
]
Rick Hillegas commented on DERBY-7006:
--
Additional complexity of solution S2:
Each jar file in the database is loaded by its own instance of
org.apache.derby.impl.services.reflect.JarLoader. Every Java ClassLoader
defines its own unnamed module. So, in addition to the catch-all unnamed module
associated with the application ClassLoader, there is an unnamed module for
each jar file in the database. It is that jar-specific unnamed module which
cannot be read by the generated module defined by solution S2. Named modules
cannot read unnamed modules, and java.lang.module.ModuleDescriptor.Builder
provides no support for allowing a generated module to access an unnamed module.
The following script demonstrates this issue.
{noformat}
connect 'jdbc:derby:memory:db;create=true';
call
sqlj.install_jar('/Users/rhillegas/derby/mainline/trunk/java/org.apache.derby.tests/org/apache/derbyTesting/functionTests/tests/store/brtestjar.jar',
'aggjar', 0);
call SYSCS_UTIL.SYSCS_SET_DATABASE_PROPERTY('derby.database.classpath',
'APP.aggjar');
create function dv(P1 INT) RETURNS INT NO SQL external name
'dbytesting.CodeInAJar.doubleMe' language java parameter style java;
-- fails because the generated module cannot read the unnamed
-- module associated with the JarLoader for APP.aggjar
values dv(3);
-- fails because the generated module cannot read the unnamed
-- module associated with the JarLoader for APP.aggjar
values dv(3) + dv(5);
{noformat}
Here is the output from running that script:
{noformat}
ij version 10.15
ij> connect 'jdbc:derby:memory:db;create=true';
ij> call
sqlj.install_jar('/Users/rhillegas/derby/mainline/trunk/java/org.apache.derby.tests/org/apache/derbyTesting/functionTests/tests/store/brtestjar.jar',
'aggjar', 0);
0 rows inserted/updated/deleted
ij> call SYSCS_UTIL.SYSCS_SET_DATABASE_PROPERTY('derby.database.classpath',
'APP.aggjar');
0 rows inserted/updated/deleted
ij> create function dv(P1 INT) RETURNS INT NO SQL external name
'dbytesting.CodeInAJar.doubleMe' language java parameter style java;
0 rows inserted/updated/deleted
ij> -- fails because the generated module cannot read the unnamed
-- module associated with the JarLoader for APP.aggjar
values dv(3);
1
---
ERROR 38000: The exception 'java.lang.IllegalAccessError: class
org.apache.derby.exe.ac3ea3c0fbx0166x6f01xaed2xe268f3f22 (in module
org.apache.derby.generatedclasses) cannot access class dbytesting.CodeInAJar
(in unnamed module @0x1c025cb) because module org.apache.derby.generatedclasses
does not read unnamed module @0x1c025cb' was thrown while evaluating an
expression.
ERROR XJ001: Java exception: 'class
org.apache.derby.exe.ac3ea3c0fbx0166x6f01xaed2xe268f3f22 (in module
org.apache.derby.generatedclasses) cannot access class dbytesting.CodeInAJar
(in unnamed module @0x1c025cb) because module org.apache.derby.generatedclasses
does not read unnamed module @0x1c025cb: java.lang.IllegalAccessError'.
ij> -- fails because the generated module cannot read the unnamed
-- module associated with the JarLoader for APP.aggjar
values dv(3) + dv(5);
1
---
ERROR 38000: The exception 'java.lang.IllegalAccessError: class
org.apache.derby.exe.ac3ea3c0fbx0166x6f01xaed2xe268f3f23 (in module
org.apache.derby.generatedclasses) cannot access class dbytesting.CodeInAJar
(in unnamed module @0x1c025cb) because module org.apache.derby.generatedclasses
does not read unnamed module @0x1c025cb' was thrown while evaluating an
expression.
ERROR XJ001: Java exception: 'class
org.apache.derby.exe.ac3ea3c0fbx0166x6f01xaed2xe268f3f23 (in module
org.apache.derby.generatedclasses) cannot access class dbytesting.CodeInAJar
(in unnamed module @0x1c025cb) because module org.apache.derby.generatedclasses
does not read unnamed module @0x1c025cb: java.lang.IllegalAccessError'.
{noformat}
> Investigate putting generated classes under the engine module loader
>
>
> Key: DERBY-7006
> URL: https://issues.apache.org/jira/browse/DERBY-7006
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.15.0.0
>Reporter: Rick Hillegas
>Priority: Major
> Attachments: derby-7006-01-aa-remiForax.diff,
> derby-7006-01-ac-alanBateman.diff
>
>
> Right now, the generated query plans are compiled into the catch-all unnamed
> module. This forces us to grant reflective access to several engine packages.
> It would be nice to encapsulate the generated classes inside the engine
> module loader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DERBY-7006) Investigate putting generated classes under the engine module loader
[ https://issues.apache.org/jira/browse/DERBY-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647919#comment-16647919 ] Rick Hillegas commented on DERBY-7006: -- Here is another issue with solution S2: Every prepared statement has its own ClassLoader (an instance of ReflectJavaLoader2). That ReflectJavaLoader2 loads the prepared statement's generated class. This lets Derby garbage-collect the generated class along with the prepared statement when the prepared statement is evicted from the statement cache. Under solution S2, the generated module is bound to the statement-specific ReflectJavaLoader2. That is why solution S2 generates one module per prepared statement. I do not see how to improve solution S2 in order to support both of the following requirements: 1) generate a single module per database 2) continue to support the garbage-collection of statements when they are evicted from the statement cache. > Investigate putting generated classes under the engine module loader > > > Key: DERBY-7006 > URL: https://issues.apache.org/jira/browse/DERBY-7006 > Project: Derby > Issue Type: Improvement > Components: SQL >Affects Versions: 10.15.0.0 >Reporter: Rick Hillegas >Priority: Major > Attachments: derby-7006-01-aa-remiForax.diff, > derby-7006-01-ac-alanBateman.diff > > > Right now, the generated query plans are compiled into the catch-all unnamed > module. This forces us to grant reflective access to several engine packages. > It would be nice to encapsulate the generated classes inside the engine > module loader. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DERBY-7006) Investigate putting generated classes under the engine module loader
[
https://issues.apache.org/jira/browse/DERBY-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646758#comment-16646758
]
Rick Hillegas commented on DERBY-7006:
--
h2. Summary of Experiments
I have put some effort into trying to get Derby to load the generated classes
into the engine module. On core-libs-...@openjdk.java.net, I started an email
thread titled "generated code and jigsaw modules". Two suggestions came back:
S1) Rémi Forax recommended that we try loading the generated bytes as follows:
{noformat}
java.lang.invoke.MethodHandles.lookup().defineClass(generatedClassBytes)
{noformat}
S2) Alan Bateman suggested that we study code from the java.xml module:
http://hg.openjdk.java.net/jdk/jdk/raw-file/tip/src/java.xml/share/classes/com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.java
I tried both approaches. See the attached patches:
derby-7006-01-aa-remiForax.diff and derby-7006-01-ac-alanBateman.diff.
Both approaches solved some important problems:
P1) running simple DDL and queries
P2) running triggers
But neither approach solved the following problem:
P3) running functions which live inside jar files stored in the database
After posting these results to core-libs-...@openjdk.java.net, I received more
kind advice, which, nevertheless, did not fix problem P3.
These are my conclusions:
C1) S1 is the simpler, more straightforward solution.
C2) S2 is more complicated to start out with. But I will play around with it
more.
Better solutions may occur as JPMS evolves. In the short-term, we could also
reduce the attack surface of the exposed engine modules. See
https://issues.apache.org/jira/browse/DERBY-7012
The following sections provide brief descriptions of the attached patches as
well as the additional context provided by my two posts to
core-libs-...@openjdk.java.net
h2. First Solution: Use MethodHandles.lookup()
The main features of this solution are:
MH1) A stub class in the generated package is added to the engine codeline.
MH2) Generated classes are loaded into the engine module by calling support
code in java.lang.invoke.MethodHandles. The support code must be called from
the stub class in order to anchor the generated classes in the generated
package:
{noformat}
java.lang.invoke.MethodHandles.lookup().defineClass(generatedClassBytes)
{noformat}
MH3) An extra permission is required for the engine jar when running under a
security manager:
{noformat}
permission java.lang.RuntimePermission "defineClass";
{noformat}
h2. Second Solution: Study TemplatesImpl
The main features of this solution are:
TI1) A separate module is created for each generated class.
TI2) The engine module exports the necessary packages to the generated module.
TI3) An extra permission is required for the engine jar when running under a
security manager:
{noformat}
permission java.lang.RuntimePermission "getProtectionDomain";
{noformat}
I need to solve the following problems:
TIP1) I don't know how to get the module names of jar files which are loaded
into the database. This could be solved if we divined the jar file names when
the jars are loaded into the database by SQLJ.INSTALL_JAR/SQLJ.REPLACE_JAR and
then stored the names in SYS.SYSTABLES.
TIP2) I think that there is a slow memory leak when the generated class is
garbage-collected and the generated module and extra exports are orphaned. I
don't know how to remove this extra garbage. But maybe we could plug the leak
by generating only one module for each database. We would need to make sure
that all of this machinery is unloaded when the embedded driver is
de-registered.
h2. First Message to core-libs-...@openjdk.java.net
I am looking for advice about how to tighten up module encapsulation while
generating byte code on the fly. I ask this question on behalf of Apache Derby,
a pure-Java relational database whose original code dates back to Java 1.2. I
want to reduce Derby's attack-surface when running with a module path.
First a little context: A relational database is an interpreter for the SQL
language. It converts SQL queries into byte code which then runs on a virtual
machine embedded in the interpreter. In Derby's case, the virtual machine is
the Java VM and the byte code is simply Java byte code. That is, a Derby query
plan is a class whose byte code is generated on the fly at run time.
I have converted the Apache Derby codeline into a set of jigsaw modules:
https://issues.apache.org/jira/browse/DERBY-6945. Unfortunately, I had to punch
holes in the encapsulation of the main Derby module so that the generated query
plans could call back into the Derby engine. That is because, by default,
generated query plans load into the catch-all, unnamed module. Note that all of
these generated classes live in a single package which does not belong to any
named module.
1) Is it possible to load
