Hello! Full Ignite logs of the problematic node will be helpful. Can you upload the log file anywhere?
Regards, -- Ilya Kasnacheev 2018-05-31 12:37 GMT+03:00 Stéphane Gayet <stephane.ga...@misterfly.com>: > Hi All, > > > Corrections about my previous email. > > > When the cluster stops responding to the sql query, I can identify a > faulting node in the following exception : > > 2018-05-31 11:20:17,036 [75] ERROR ServiceCache - Failed to get RtProposal > > Apache.Ignite.Core.Cache.CacheException: Failed to run map query > remotely.Failed to execute map query on the node: > *90bd677d-dee5-44bb-af6f-80786b85bd37*, class org.apache.ignite.internal. > processors.query.IgniteSQLException:Failed to set schema for DB > connection for thread [schema=owproposals] ---> > Apache.Ignite.Core.Common.JavaException: > javax.cache.CacheException: Failed to run map query remotely.Failed to > execute map query on the node: 90bd677d-dee5-44bb-af6f-80786b85bd37, > class org.apache.ignite.internal.processors.query.IgniteSQLException:Failed > to set schema for DB connection for thread [schema=owproposals] > > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:747) > > at org.apache.ignite.internal.processors.query.h2. > IgniteH2Indexing$8.iterator(IgniteH2Indexing.java:1339) > > at org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator( > QueryCursorImpl.java:95) > > at org.apache.ignite.internal.processors.platform.cache.query. > PlatformAbstractQueryCursor.processInLongOutLong( > PlatformAbstractQueryCursor.java:147) > > at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl. > inLongOutLong(PlatformTargetProxyImpl.java:55) > > Caused by: javax.cache.CacheException: Failed to execute map query on the > node: 90bd677d-dee5-44bb-af6f-80786b85bd37, class > org.apache.ignite.internal.processors.query.IgniteSQLException:Failed to > set schema for DB connection for thread [schema=owproposals] > > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:275) > > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.onFail(GridReduceQueryExecutor.java:265) > > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.onMessage(GridReduceQueryExecutor.java:244) > > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor$2.onMessage(GridReduceQueryExecutor.java:188) > > at org.apache.ignite.internal.managers.communication. > GridIoManager$ArrayListener.onMessage(GridIoManager.java:2332) > > at org.apache.ignite.internal.managers.communication. > GridIoManager.invokeListener(GridIoManager.java:1555) > > at org.apache.ignite.internal.managers.communication.GridIoManager. > processRegularMessage0(GridIoManager.java:1183) > > at org.apache.ignite.internal.managers.communication. > GridIoManager.access$4200(GridIoManager.java:126) > > at org.apache.ignite.internal.managers.communication.GridIoManager$9.run( > GridIoManager.java:1090) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > à Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.ExceptionCheck() > > à Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.CallLongMethod(GlobalRef > obj, IntPtr methodId, Int64* argsPtr) > > à > Apache.Ignite.Core.Impl.Unmanaged.UnmanagedUtils.TargetInLongOutLong(GlobalRef > target, Int32 opType, Int64 memPtr) > > à Apache.Ignite.Core.Impl.PlatformJniTarget.InLongOutLong(Int32 type, > Int64 val) > > --- Fin de la trace de la pile d'exception interne --- > > à Apache.Ignite.Core.Impl.PlatformJniTarget.InLongOutLong(Int32 type, > Int64 val) > > à Apache.Ignite.Core.Impl.Cache.Query.QueryCursorBase`1.GetEnumerator() > > à MrFly.CacheDlm.Common.Services.ServiceCache. > BuildRtProposalFromOwProposal(String enterprise, String route1, String > route2, DateTime departureDate, DateTime returnDate, Int32 nbAdt, Int32 > nbChd, Int32 nbInf) dans C:\DevRoot\A.Ignite\MrFly. > CacheDlm.Common\Services\ServiceCache.cs:ligne 1015 > > à MrFly.CacheDlm.Common.Services.ServiceCache.BuildRtProposal(String > enterprise, String route1, String route2, DateTime departureDate, DateTime > returnDate, Int32 nbAdt, Int32 nbChd, Int32 nbInf) dans > C:\DevRoot\A.Ignite\MrFly.CacheDlm.Common\Services\ServiceCache.cs:ligne > 913 > > à MrFly.CacheDlm.Common.Services.ServiceCache.GetRtProposal(String > enterprise, String route1, String route2, DateTime departureDate, DateTime > returnDate, Int32 nbAdt, Int32 nbChd, Int32 nbInf, Boolean withCache) dans > C:\DevRoot\A.Ignite\MrFly.CacheDlm.Common\Services\ServiceCache.cs:ligne > 646 > > > If I stop Ignite on this node, the cluster starts responding again. > > > So my questions are : why the node stops responding? How to identify root > causes? > > > Regards, > > Stephane Gayet > > > > ------------------------------ > *De :* Stéphane Gayet <stephane.ga...@misterfly.com> > *Envoyé :* mercredi 30 mai 2018 23:44 > *À :* user@ignite.apache.org > *Objet :* SQL Query error > > > Hi all, > > > We have installed a 2.4 Ignite cluster > > - 3 nodes under Windows systems (24 Go, 16 Go, 16 Go) > > - 4 caches configured, partitioned, no backup > > - no persistence > > > cache @c0 contains around 60,000 items, > > cache @c3 contains few items (around 200) but items are very large. > > > We run sql queries which aggregate the @c0 items in large collections > (until 14,000 items per collection) and store the result in @c3. > > > After a while, the sql query stop functionning. The following error is > logged : > > > 2018-05-30 23:14:42,716 [282] ERROR org.apache.ignite.internal. > processors.query.h2.twostep.GridMapQueryExecutor - Failed to execute > local query. > > > Here is our cache configuration : > > <cacheConfiguration> > <cacheConfiguration name="owproposals" cacheMode="Partitioned" > backups="0" readThrough="true" writeThrough="true" > writeBehindEnabled="true"> > <cacheStoreFactory type="OwProposalFactory"/> > <queryEntities> > <queryEntity valueType="OwProposal, Cache.Common" > valueTypeName="Models.OwProposal"/> > </queryEntities> > </cacheConfiguration> > <cacheConfiguration name="rtproposals" cacheMode="Partitioned" > backups="0" readThrough="true" writeThrough="true" > writeBehindEnabled="true"> > <cacheStoreFactory type="RtProposalFactory"/> > <queryEntities> > <queryEntity valueType="RtProposal, Cache.Common" > valueTypeName="Models.RtProposal"/> > </queryEntities> > </cacheConfiguration> > <cacheConfiguration name="owcollection" cacheMode="Partitioned" > backups="0"> > <queryEntities> > <queryEntity valueType="OwCollection, Cache.Common" > valueTypeName="Models.OwCollection"/> > </queryEntities> > </cacheConfiguration> > <cacheConfiguration name="rtcollection" cacheMode="Partitioned" > backups="0"> > <queryEntities> > <queryEntity valueType="RtCollection, Cache.Common" > valueTypeName="Models.RtCollection"/> > </queryEntities> > </cacheConfiguration> > </cacheConfiguration> > > > I tried to clear the items of @c0 cache before re-populate it but I got > the error : > > 2018-05-30 23:34:34,333 [16] ERROR ServiceCache - Failed to delete OwItems > older than 2018-05-31 > Apache.Ignite.Core.Common.IgniteException: Failed to execute map query on > the node: b9f240d6-0ee6-4dee-bda0-51088a743481, class > org.apache.ignite.internal.processors.query.IgniteSQLException:Failed to > set schema for DB connection for thread [schema=owproposals] ---> > Apache.Ignite.Core.Common.JavaException: class > org.apache.ignite.IgniteCheckedException: > Failed to execute map query on the node: b9f240d6-0ee6-4dee-bda0-51088a743481, > class org.apache.ignite.internal.processors.query.IgniteSQLException:Failed > to set schema for DB connection for thread [schema=owproposals] > at org.apache.ignite.internal.processors.platform.utils.PlatformUtils. > unwrapQueryException(PlatformUtils.java:519) > at org.apache.ignite.internal.processors.platform.cache.query. > PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor. > java:132) > at org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl. > outStream(PlatformTargetProxyImpl.java:93) > Caused by: javax.cache.CacheException: Failed to run map query > remotely.Failed to execute map query on the node: > b9f240d6-0ee6-4dee-bda0-51088a743481, > class org.apache.ignite.internal.processors.query.IgniteSQLException:Failed > to set schema for DB connection for thread [schema=owproposals] > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:747) > at org.apache.ignite.internal.processors.query.h2. > IgniteH2Indexing$8.iterator(IgniteH2Indexing.java:1339) > at org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator( > QueryCursorImpl.java:95) > at org.apache.ignite.internal.processors.query.h2. > IgniteH2Indexing$9.iterator(IgniteH2Indexing.java:1403) > at org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator( > QueryCursorImpl.java:95) > at org.apache.ignite.internal.processors.cache.QueryCursorImpl.getAll( > QueryCursorImpl.java:127) > at org.apache.ignite.internal.processors.platform.cache.query. > PlatformAbstractQueryCursor.processOutStream(PlatformAbstractQueryCursor. > java:127) > ... 1 more > Caused by: javax.cache.CacheException: Failed to execute map query on the > node: b9f240d6-0ee6-4dee-bda0-51088a743481, class > org.apache.ignite.internal.processors.query.IgniteSQLException:Failed to > set schema for DB connection for thread [schema=owproposals] > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:275) > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.onFail(GridReduceQueryExecutor.java:265) > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor.onMessage(GridReduceQueryExecutor.java:244) > at org.apache.ignite.internal.processors.query.h2.twostep. > GridReduceQueryExecutor$2.onMessage(GridReduceQueryExecutor.java:188) > at org.apache.ignite.internal.managers.communication. > GridIoManager$ArrayListener.onMessage(GridIoManager.java:2332) > at org.apache.ignite.internal.managers.communication. > GridIoManager.invokeListener(GridIoManager.java:1555) > at org.apache.ignite.internal.managers.communication.GridIoManager. > processRegularMessage0(GridIoManager.java:1183) > at org.apache.ignite.internal.managers.communication. > GridIoManager.access$4200(GridIoManager.java:126) > at org.apache.ignite.internal.managers.communication.GridIoManager$9.run( > GridIoManager.java:1090) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > > à Apache.Ignite.Core.Impl.Unmanaged.Jni.Env.ExceptionCheck() > à > Apache.Ignite.Core.Impl.Unmanaged.UnmanagedUtils.TargetOutStream(GlobalRef > target, Int32 opType, Int64 memPtr) > à Apache.Ignite.Core.Impl.PlatformJniTarget.OutStream[T](Int32 type, > Func`2 readAction) > --- Fin de la trace de la pile d'exception interne --- > à Apache.Ignite.Core.Impl.PlatformJniTarget.OutStream[T](Int32 type, > Func`2 readAction) > à Apache.Ignite.Core.Impl.Cache.Query.QueryCursorBase`1.GetAll() > à MrFly.CacheDlm.Common.Services.ServiceCache.GetKeys[TV](QueryBase > query) > à MrFly.CacheDlm.Common.Services.ServiceCache.DeleteOneWayItems(DateTime > createdDate) > > > At this time, the only way is to down the three nodes and restart them > from scratch. > > Any idea about what is malfunctionning or misconfigured ? > > Kind regards, > > S Gayet > > > > >
