RE: Re: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite
When I got OutOfMemory and become research heapDumps, I have found some bugs and leaks not only in Ignite but in our code too. So I already implement some fixes in our system, and I don’t see this error more. May be it was because of our logic (we register/unregister drivers too). So if I get it again, I will submit reproduce scenario. Andrey. От: Ilya KasnacheevОтправлено: 25 марта 2020 г. в 15:37Кому: user@ignite.apache.orgТема: Re: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite Hello! I have never seen a stack trace like this one. Can you provide a reproducer for this behavior? Regards,-- Ilya Kasnacheev пт, 20 мар. 2020 г. в 20:20, Andrey Davydov <andrey.davy...@gmail.com>:Hello, Current implementation is really unsafe for multiple Ignite in same JVM.In tests for our system when we stop/start nodes in different order we get following error: Caused by: class org.apache.ignite.IgniteCheckedException: Failed to initialize system DB connection: jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngineat org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1402)at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1035)at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:921)at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:820)at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690)at com.example.testutils.TestNode.start(TestNode.java:75)... 38 moreCaused by: class org.apache.ignite.internal.processors.query.IgniteSQLException: Failed to initialize system DB connection: jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngineat org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:213)at org.apache.ignite.internal.processors.query.h2.ConnectionManager.(ConnectionManager.java:152)at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.start(IgniteH2Indexing.java:2070)at org.apache.ignite.internal.processors.query.GridQueryProcessor.start(GridQueryProcessor.java:256)at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1978)at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1212)... 46 moreCaused by: java.sql.SQLException: No suitable driver found for jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngineat java.sql.DriverManager.getConnection(DriverManager.java:689)at java.sql.DriverManager.getConnection(DriverManager.java:270)at org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:206)... 51 more On Thu, Mar 19, 2020 at 6:52 PM Andrey Davydov <andrey.davy...@gmail.com> wrote:It seems like moving in right way =) Let wait for release. Andrey. От: Andrey MashenkovОтправлено: 19 марта 2020 г. в 16:28Кому: user@ignite.apache.orgТема: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite Hi, In Apache Ignite master branch I see a separate class H2JavaObjectSerializer that implements JavaObjectSerializer.Seems, this won't be released in 2.8https://issues.apache.org/jira/browse/IGNITE-12609 On Thu, Mar 19, 2020 at 4:03 PM Andrey Davydov <andrey.davy...@gmail.com> wrote:Seem that refactor h2serilizer method in following manner will be safe for marshallers which not depends on ignite instance and will be faster anyway, due to single clsLdr resolving. For binary marshaller solution is still unsafe =((( private JavaObjectSerializer h2Serializer() { ClassLoader clsLdr = ctx != null ? U.resolveClassLoader(ctx.config()) : null; return new CustomJavaObjectSerializer(marshaller, clsLdr); } private static f
Re: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite
Hello!
I have never seen a stack trace like this one. Can you provide a reproducer
for this behavior?
Regards,
--
Ilya Kasnacheev
пт, 20 мар. 2020 г. в 20:20, Andrey Davydov :
> Hello,
>
> Current implementation is really unsafe for multiple Ignite in same JVM.
> In tests for our system when we stop/start nodes in different order we get
> following error:
>
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> initialize system DB connection:
> jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1402)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
> at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1035)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:921)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:820)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690)
> at com.example.testutils.TestNode.start(TestNode.java:75)
> ... 38 more
> Caused by: class
> org.apache.ignite.internal.processors.query.IgniteSQLException: Failed to
> initialize system DB connection:
> jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine
> at
> org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:213)
> at
> org.apache.ignite.internal.processors.query.h2.ConnectionManager.(ConnectionManager.java:152)
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.start(IgniteH2Indexing.java:2070)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.start(GridQueryProcessor.java:256)
> at
> org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1978)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1212)
> ... 46 more
> Caused by: java.sql.SQLException: No suitable driver found for
> jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine
> at java.sql.DriverManager.getConnection(DriverManager.java:689)
> at java.sql.DriverManager.getConnection(DriverManager.java:270)
> at
> org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:206)
> ... 51 more
>
>
> On Thu, Mar 19, 2020 at 6:52 PM Andrey Davydov
> wrote:
>
>> It seems like moving in right way =) Let wait for release.
>>
>>
>>
>> Andrey.
>>
>>
>>
>> *От: *Andrey Mashenkov
>> *Отправлено: *19 марта 2020 г. в 16:28
>> *Кому: *user@ignite.apache.org
>> *Тема: *Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite
>>
>>
>>
>> Hi,
>>
>>
>>
>> In Apache Ignite master branch I see a separate class
>> H2JavaObjectSerializer that implements JavaObjectSerializer.
>>
>> Seems, this won't be released in 2.8
>>
>> https://issues.apache.org/jira/browse/IGNITE-12609
>>
>>
>>
>> On Thu, Mar 19, 2020 at 4:03 PM Andrey Davydov
>> wrote:
>>
>> Seem that refactor h2serilizer method in following manner will be safe
>> for marshallers which not depends on ignite instance and will be faster
>> anyway, due to single clsLdr resolving. For binary marshaller solution is
>> still unsafe =(((
>>
>>
>>
>> private JavaObjectSerializer h2Serializer() {
>>
>> ClassLoader clsLdr = ctx != null ?
>> U.resolveClassLoader(ctx.config()) : null;
Re: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite
Hello,
Current implementation is really unsafe for multiple Ignite in same JVM.
In tests for our system when we stop/start nodes in different order we get
following error:
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
initialize system DB connection:
jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1402)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1035)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:921)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:820)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690)
at com.example.testutils.TestNode.start(TestNode.java:75)
... 38 more
Caused by: class
org.apache.ignite.internal.processors.query.IgniteSQLException: Failed to
initialize system DB connection:
jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine
at
org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:213)
at
org.apache.ignite.internal.processors.query.h2.ConnectionManager.(ConnectionManager.java:152)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.start(IgniteH2Indexing.java:2070)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.start(GridQueryProcessor.java:256)
at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1978)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1212)
... 46 more
Caused by: java.sql.SQLException: No suitable driver found for
jdbc:h2:mem:fd191fac-c2f1-4398-bf8a-0dcddf651830;LOCK_MODE=3;MULTI_THREADED=1;DB_CLOSE_ON_EXIT=FALSE;DEFAULT_LOCK_TIMEOUT=1;FUNCTIONS_IN_SCHEMA=true;OPTIMIZE_REUSE_RESULTS=0;QUERY_CACHE_SIZE=0;MAX_OPERATION_MEMORY=0;BATCH_JOINS=1;ROW_FACTORY="org.apache.ignite.internal.processors.query.h2.opt.H2PlainRowFactory";DEFAULT_TABLE_ENGINE=org.apache.ignite.internal.processors.query.h2.opt.GridH2DefaultTableEngine
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
org.apache.ignite.internal.processors.query.h2.ConnectionManager.connectionNoCache(ConnectionManager.java:206)
... 51 more
On Thu, Mar 19, 2020 at 6:52 PM Andrey Davydov
wrote:
> It seems like moving in right way =) Let wait for release.
>
>
>
> Andrey.
>
>
>
> *От: *Andrey Mashenkov
> *Отправлено: *19 марта 2020 г. в 16:28
> *Кому: *user@ignite.apache.org
> *Тема: *Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite
>
>
>
> Hi,
>
>
>
> In Apache Ignite master branch I see a separate class
> H2JavaObjectSerializer that implements JavaObjectSerializer.
>
> Seems, this won't be released in 2.8
>
> https://issues.apache.org/jira/browse/IGNITE-12609
>
>
>
> On Thu, Mar 19, 2020 at 4:03 PM Andrey Davydov
> wrote:
>
> Seem that refactor h2serilizer method in following manner will be safe for
> marshallers which not depends on ignite instance and will be faster anyway,
> due to single clsLdr resolving. For binary marshaller solution is still
> unsafe =(((
>
>
>
> private JavaObjectSerializer h2Serializer() {
>
> ClassLoader clsLdr = ctx != null ?
> U.resolveClassLoader(ctx.config()) : null;
>
> return new CustomJavaObjectSerializer(marshaller, clsLdr);
>
> }
>
>
>
> private static final class CustomJavaObjectSerializer implements
> JavaObjectSerializer {
>
> private final Marshaller marshaller;
>
> private final ClassLoader clsLdr;
>
>
>
> CustomJavaObjectSerializer(Marshaller marshaller, ClassLoader
> clsLdr) {
>
> this.marshaller = marshaller;
>
> this.clsLdr = clsLdr;
>
> }
>
>
>
> @Override public byte[] serialize(Obj
RE: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite
It seems like moving in right way =) Let wait for release. Andrey. От: Andrey MashenkovОтправлено: 19 марта 2020 г. в 16:28Кому: user@ignite.apache.orgТема: Re: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite Hi, In Apache Ignite master branch I see a separate class H2JavaObjectSerializer that implements JavaObjectSerializer.Seems, this won't be released in 2.8https://issues.apache.org/jira/browse/IGNITE-12609 On Thu, Mar 19, 2020 at 4:03 PM Andrey Davydov <andrey.davy...@gmail.com> wrote:Seem that refactor h2serilizer method in following manner will be safe for marshallers which not depends on ignite instance and will be faster anyway, due to single clsLdr resolving. For binary marshaller solution is still unsafe =((( private JavaObjectSerializer h2Serializer() { ClassLoader clsLdr = ctx != null ? U.resolveClassLoader(ctx.config()) : null; return new CustomJavaObjectSerializer(marshaller, clsLdr); } private static final class CustomJavaObjectSerializer implements JavaObjectSerializer { private final Marshaller marshaller; private final ClassLoader clsLdr; CustomJavaObjectSerializer(Marshaller marshaller, ClassLoader clsLdr) { this.marshaller = marshaller; this.clsLdr = clsLdr; } @Override public byte[] serialize(Object obj) throws Exception { return U.marshal(marshaller, obj); } @Override public Object deserialize(byte[] bytes) throws Exception { return U.unmarshal(marshaller, bytes, clsLdr); } } Andrey. От: Andrey DavydovОтправлено: 19 марта 2020 г. в 15:43Кому: user@ignite.apache.orgТема: RE: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite I have some RnD with Apache Felix this week to found workaround for multi-tenancy of H2. But there is problem with some Ignites in same JVM. As I see in org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing latest started Ignite will visible via JdbcUtils.serializer, and it can be already closed and it workdir can be deleted. Line 2105: if (JdbcUtils.serializer != null) U.warn(log, "Custom H2 serialization is already configured, will override."); JdbcUtils.serializer = h2Serializer(); Line 2268: private JavaObjectSerializer h2Serializer() { return new JavaObjectSerializer() { //nested class has link to parent IgniteH2Indexing and to ingnite instance transitively @Override public byte[] serialize(Object obj) throws Exception { return U.marshal(marshaller, obj); //In common case, binary marshaller logic depends on work dir } @Override public Object deserialize(byte[] bytes) throws Exception { ClassLoader clsLdr = ctx != null ? U.resolveClassLoader(ctx.config()) : null; //only configuration need, but all ctx leaked return U.unmarshal(marshaller, bytes, clsLdr); } }; } Andrey. От: Ilya KasnacheevОтправлено: 19 марта 2020 г. в 14:37Кому: user@ignite.apache.orgТема: Re: Unsafe usage of org.h2.util.JdbcUtils in Ignite Hello! As far as my understanding goes: 1) It is H2's decision to exhibit JdbcUtil.serializer as their public API, they have a public system property to override it: /** * System property h2.javaObjectSerializer * (default: null). * The JavaObjectSerializer class name for java objects being stored in * column of type OTHER. It must be the same on client and server to work * correctly. */public static final String JAVA_OBJECT_SERIALIZER = Utils.getProperty("h2.javaObjectSerializer", null); Obviously, this is not designed for multi-tenancy of H2 in mind. If you really need multi-tenancy, I recommend starting H2 in a separate class loader inherited from root class loader and isolated from any Ignite classes. Regards,-- Ilya Kasnacheev ср, 18 мар. 2020 г. в 18:54, Andrey Davydov <andrey.davy...@gmail.com>:Hello, org.h2.util.JdbcUtils is utility class with all static methods and configured via System.properties. So it system wide resource. It is incorrect inject Ignite specific settings in it. this - value: org.apache.ignite.internal.IgniteKernal #1<- grid - class: org.apache.ignite.internal.GridKernalContextImpl, value: org.apache.ignite.internal.IgniteKernal #1 <- ctx - class: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing, value: org.apache.ignite.internal.GridKernalContextImpl #2 <- this$0 - class: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$10, value: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing #2 <- serializer - class: org.h2.util.JdbcUtils, value: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$10 #1 <- [5395] - class: java.lang.Object[], value: org.h2.util.JdbcUtils class JdbcUtils <- elementData - class: java.util.Vector, value: java.lang.Object[] #37309 <- classes
