RE: Re: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite

[Andrey Davydov]

When I got OutOfMemory and become research heapDumps, I have found some bugs and leaks not only in Ignite but in our code too. So I already implement some fixes in our system, and I don’t see this error more. May be it was because of our logic (we register/unregister drivers too). So if I get it again, I will submit reproduce scenario.   Andrey.   От: Ilya Kasnacheev Отправлено: 25 марта 2020 г. в 15:37 Кому: user@ignite.apache.org Тема: Re: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite   Hello!   I have never seen a stack trace like this one. Can you provide a reproducer for this behavior?   Regards, -- Ilya Kasnacheev     пт, 20 мар. 2020 г. в 20:20, Andrey Davydov <andrey.davy...@gmail.com>: Hello,   Current implementation is really unsafe for multiple Ignite in same JVM. In tests for our system when we stop/start nodes in different order we get following error:   Caused by: class org.apache.ignite.IgniteCheckedException: Failed to initialize system DB connection: jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=10000;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1402) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1035) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:921) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:820) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690) at com.example.testutils.TestNode.start(TestNode.java:75) ... 38 more Caused by: class org.apache.ignite.internal.processors.query.IgniteSQLException: Failed to initialize system DB connection: jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=10000;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine at org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:213) at org.apache.ignite.internal.processors.query.h2.ConnectionManager.<init>(ConnectionManager.java:152) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.start(IgniteH2Indexing.java:2070) at org.apache.ignite.internal.processors.query.GridQueryProcessor.start(GridQueryProcessor.java:256) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1978) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1212) ... 46 more Caused by: java.sql.SQLException: No suitable driver found for jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=10000;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine at java.sql.DriverManager.getConnection(DriverManager.java:689) at java.sql.DriverManager.getConnection(DriverManager.java:270) at org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:206) ... 51 more     On Thu, Mar 19, 2020 at 6:52 PM Andrey Davydov <andrey.davy...@gmail.com> wrote: It seems like moving in right way =) Let wait for release.   Andrey.   От: Andrey Mashenkov Отправлено: 19 марта 2020 г. в 16:28 Кому: user@ignite.apache.org Тема: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite   Hi,   In Apache Ignite master branch I see a separate class H2JavaObjectSerializer that implements JavaObjectSerializer. Seems, this won't be released in 2.8 https://issues.apache.org/jira/browse/IGNITE-12609   On Thu, Mar 19, 2020 at 4:03 PM Andrey Davydov <andrey.davy...@gmail.com> wrote: Seem that refactor h2serilizer method in following manner will be safe for marshallers which not depends on ignite instance and will be faster anyway, due to single clsLdr resolving. For binary marshaller solution is still unsafe =(((         private JavaObjectSerializer h2Serializer() {         ClassLoader clsLdr = ctx != null ? U.resolveClassLoader(ctx.config()) : null;         return new CustomJavaObjectSerializer(marshaller, clsLdr);     }       private static final class CustomJavaObjectSerializer implements JavaObjectSerializer {         private final Marshaller marshaller;         private final ClassLoader clsLdr;                 CustomJavaObjectSerializer(Marshaller marshaller, ClassLoader clsLdr) {             this.marshaller = marshaller;             this.clsLdr = clsLdr;         }                 @Override public byte[] serialize(Object obj) throws Exception {             return U.marshal(marshaller, obj);         }           @Override public Object deserialize(byte[] bytes) throws Exception {             return U.unmarshal(marshaller, bytes, clsLdr);         }     }   Andrey.   От: Andrey Davydov Отправлено: 19 марта 2020 г. в 15:43 Кому: user@ignite.apache.org Тема: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite   I have some RnD with Apache Felix this week to found workaround for multi-tenancy of H2.   But there is problem with some Ignites in same JVM.   As I see in org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing latest started Ignite will visible via JdbcUtils.serializer, and it can be already closed and it workdir can be deleted.   Line 2105:             if (JdbcUtils.serializer != null)             U.warn(log, "Custom H2 serialization is already configured, will override.");           JdbcUtils.serializer = h2Serializer();   Line 2268:       private JavaObjectSerializer h2Serializer() {         return new JavaObjectSerializer() {  //nested class has link to parent IgniteH2Indexing and to ingnite instance transitively             @Override public byte[] serialize(Object obj) throws Exception {                 return U.marshal(marshaller, obj); //In common case, binary marshaller logic depends on work dir             }               @Override public Object deserialize(byte[] bytes) throws Exception {                 ClassLoader clsLdr = ctx != null ? U.resolveClassLoader(ctx.config()) : null; //only configuration need, but all ctx leaked                   return U.unmarshal(marshaller, bytes, clsLdr);             }         };     }     Andrey.   От: Ilya Kasnacheev Отправлено: 19 марта 2020 г. в 14:37 Кому: user@ignite.apache.org Тема: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite   Hello!   As far as my understanding goes:   1) It is H2's decision to exhibit JdbcUtil.serializer as their public API, they have a public system property to override it:   /** * System property <code>h2.javaObjectSerializer</code> * (default: null).<br /> * The JavaObjectSerializer class name for java objects being stored in * column of type OTHER. It must be the same on client and server to work * correctly. */ public static final String JAVA_OBJECT_SERIALIZER =         Utils.getProperty("h2.javaObjectSerializer", null);   Obviously, this is not designed for multi-tenancy of H2 in mind.   If you really need multi-tenancy, I recommend starting H2 in a separate class loader inherited from root class loader and isolated from any Ignite classes.   Regards, -- Ilya Kasnacheev     ср, 18 мар. 2020 г. в 18:54, Andrey Davydov <andrey.davy...@gmail.com>: Hello,   org.h2.util.JdbcUtils is utility class with all static methods  and configured via System.properties. So it system wide resource. It is incorrect inject Ignite specific settings in it.   this     - value: org.apache.ignite.internal.IgniteKernal #1 <- grid     - class: org.apache.ignite.internal.GridKernalContextImpl, value: org.apache.ignite.internal.IgniteKernal #1   <- ctx     - class: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing, value: org.apache.ignite.internal.GridKernalContextImpl #2    <- this$0     - class: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$10, value: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing #2     <- serializer     - class: org.h2.util.JdbcUtils, value: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$10 #1      <- [5395]     - class: java.lang.Object[], value: org.h2.util.JdbcUtils class JdbcUtils       <- elementData     - class: java.util.Vector, value: java.lang.Object[] #37309        <- classes     - class: sun.misc.Launcher$AppClassLoader, value: java.util.Vector #31         <- contextClassLoader (thread object)     - class: java.lang.Thread, value: sun.misc.Launcher$AppClassLoader #1   1.It cause problems, if it need to work with H2 databases from same JVM where ignite run. 2.It cause problems, when some Ignites run in same JVM 3.It makes closed IgniteKernal reachable from GC root.   I think it is bad architecture solution to use this class and use H2 related system properties at all.   Andrey.         -- Best regards, Andrey V. Mashenkov