Hi Michael, do you have any update for the issue? On Thu, Nov 2, 2017 at 5:14 PM, Michael Cherkasov < michael.cherka...@gmail.com> wrote:
> Hi Tobias, > > Thank you for explaining how to reproduce it, I'll try your instruction. I > spend several days trying to reproduce the issue, > but I thought that the reason of this is too high load and I didn't stop > client during testing. > I'll check your instruction and try to fix the issue. > > Thanks, > Mike. > > 2017-10-25 16:23 GMT+03:00 Tobias Eriksson <tobias.eriks...@qvantel.com>: > >> Hi Andrey et al >> >> I believe I now know what the problem is, the Cassandra session is >> refreshed, but before it is a prepared statement is created/used and there, >> and so using a new session with an old prepared statement is not working. >> >> >> >> The way to reproduce is >> >> 1) Start Ignite Server Node >> >> 2) Start client which inserts a batch of 100 elements >> >> 3) End client >> >> 4) Now Ignite Server Node returns the Cassandra Session to the pool >> >> 5) Wait 5+ minutes >> >> 6) Now Ignite Server Node has does a clean-up of the “unused” >> Cassandra sessions >> >> 7) Start client which inserts a batch of 100 elements >> >> 8) Boom ! The exception starts to happen >> >> >> >> Reason is >> >> 1) Execute is called for a BATCH >> >> 2) Prepared-statement is reused since there is a cache of those >> >> 3) It is about to do session().execute( batch ) >> >> 4) BUT the call to session() results in refreshing the session, >> and this is where the prepared statements as the old session new them are >> cleaned up >> >> 5) Now it is looping over 100 times with a NEW session but with an >> OLD prepared statement >> >> >> >> This is a bug, >> >> >> >> -Tobias >> >> >> >> >> >> *From: *Andrey Mashenkov <andrey.mashen...@gmail.com> >> *Reply-To: *"user@ignite.apache.org" <user@ignite.apache.org> >> *Date: *Wednesday, 25 October 2017 at 14:12 >> *To: *"user@ignite.apache.org" <user@ignite.apache.org> >> *Subject: *Re: Ignite-cassandra module issue >> >> >> >> Hi Tobias, >> >> >> >> What ignite version do you use? May be this was already fixed in latest >> one? >> >> I see related fix inclueded in upcoming 2.3 version. >> >> >> >> See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end >> logic, so session should be closed in proper way. >> >> >> >> [1] https://issues.apache.org/jira/browse/IGNITE-5897 >> >> >> >> >> >> On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson < >> tobias.eriks...@qvantel.com> wrote: >> >> Hi >> Sorry did not include the context when I replied >> Has anyone been able to resolve this problem, cause I have it too on and >> off >> In fact it sometimes happens just like that, e.g. I have been running my >> Ignite client and then stop it, and then it takes a while and run it >> again, >> and all by a sudden this error shows up. An that is the first thing that >> happens, and there is NOT a massive amount of load on Cassandra at that >> time. But I have also seen it when I hammer Ignite/Cassandra with >> updates/inserts. >> >> This is a deal-breaker for me, I need to understand how to fix this, cause >> having this in production is not an option. >> >> -Tobias >> >> >> Hi! >> I'm using the cassandra as persistence store for my caches and have one >> issue by handling a huge data (via IgniteDataStreamer from kafka). >> Ignite Configuration: >> final IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); >> igniteConfiguration.setIgniteInstanceName("test"); >> igniteConfiguration.setClientMode(true); >> igniteConfiguration.setGridLogger(new Slf4jLogger()); >> igniteConfiguration.setMetricsLogFrequency(0); >> igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi()); >> final BinaryConfiguration binaryConfiguration = new BinaryConfiguration(); >> binaryConfiguration.setCompactFooter(false); >> igniteConfiguration.setBinaryConfiguration(binaryConfiguration); >> igniteConfiguration.setPeerClassLoadingEnabled(true); >> final MemoryPolicyConfiguration memoryPolicyConfiguration = new >> MemoryPolicyConfiguration(); >> memoryPolicyConfiguration.setName("3Gb_Region_Eviction"); >> memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L); >> memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L); >> >> memoryPolicyConfiguration.setPageEvictionMode(DataPageEvicti >> onMode.RANDOM_2_LRU); >> final MemoryConfiguration memoryConfiguration = new MemoryConfiguration(); >> memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration); >> igniteConfiguration.setMemoryConfiguration(memoryConfiguration); >> >> Cache configuration: >> final CacheConfiguration<String, BinaryObject> cacheConfiguration = new >> CacheConfiguration<>(); >> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC); >> cacheConfiguration.setStoreKeepBinary(true); >> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); >> cacheConfiguration.setBackups(0); >> cacheConfiguration.setStatisticsEnabled(false); >> cacheConfiguration.setName("TestCache"); >> >> cacheConfiguration.setReadThrough(true); >> cacheConfiguration.setWriteThrough(true); >> >> cacheConfiguration.setWriteBehindEnabled(true); >> cacheConfiguration.setWriteBehindFlushFrequency(10000); >> cacheConfiguration.setWriteBehindFlushSize(0); >> cacheConfiguration.setWriteBehindFlushThreadCount(2); >> cacheConfiguration.setWriteBehindBatchSize(10000); >> >> >> final CassandraCacheStoreFactory<String, BinaryObject> >> cacheStoreFactory = new CassandraCacheStoreFactory<>(); >> final DataSource dataSource = new DataSource(); >> dataSource.setContactPoints(contactPoints); >> dataSource.setReadConsistency("ONE"); >> dataSource.setWriteConsistency("ONE"); >> dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new >> RoundRobinPolicy())); >> cacheStoreFactory.setDataSource(dataSource); >> >> final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\" >> table=\"%s\">" + >> "<keyPersistence class=\"%s\" strategy=\"%s\"/>" + >> "<valuePersistence class=\"%s\" strategy=\"%s\"/>" + >> "</persistence>"; >> final KeyValuePersistenceSettings settings = new >> KeyValuePersistenceSettings( >> String.format(CASSANDRA_PERSISTENCE, "test", >> "test_table", >> "java.lang.String", "PRIMITIVE", >> "org.apache.ignite.binary.BinaryObject", >> "BLOB")); >> cacheStoreFactory.setPersistenceSettings(settings); >> cacheConfiguration.setCacheStoreFactory(cacheStoreFactory); >> >> When application works some time (hour or more, may be less - from time to >> time) I see this exceptions on the ignite nodes: >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072] >> ^-- H/N/C [hosts=3, nodes=3, CPUs=96] >> ^-- CPU [cur=0%, avg=1.86%, GC=0%] >> ^-- PageMemory [pages=118064] >> ^-- Heap [used=4800MB, free=53.12%, comm=10240MB] >> ^-- Non heap [used=78MB, free=-1%, comm=80MB] >> ^-- Public thread pool [active=0, idle=32, qSize=0] >> ^-- System thread pool [active=0, idle=32, qSize=0] >> ^-- Outbound messages queue [size=0] >> [15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] >> FreeList >> [name=null, buckets=256, dataPages=102080, reusePages=0] >> [15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared >> statement cluster error detected, refreshing Cassandra session >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra >> session refreshed >> [15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for >> 332ms >> [15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for >> 332ms >> [15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared >> statement cluster error detected, refreshing Cassandra session >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed >> [15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed >> [15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed >> [15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed >> [15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed >> [15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed >> [15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed >> [15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 >> attempt >> to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed >> [15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed >> [15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2 >> attempt to execute Cassandra CQL statement: select "value" from >> "triggers"."callback" where "key"=?; >> [15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra >> session refreshed >> [15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for >> 229ms >> [15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared >> statement cluster error detected, another thread already fixed the problem >> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >> execute >> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have >> used a PreparedStatement that was created with another Cluster instance. >> at >> com.datastax.driver.core.SessionManager.makeRequestMessage(S >> essionManager.java:568) >> at >> com.datastax.driver.core.SessionManager.executeAsync(Session >> Manager.java:131) >> at >> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) >> at >> org.apache.ignite.cache.store.cassandra.session.CassandraSes >> sionImpl.execute(CassandraSessionImpl.java:144) >> at >> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >> load(CassandraCacheStore.java:168) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheW >> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >> at >> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >> pter.java:416) >> at >> org.apache.ignite.internal.processors.cache.store.GridCacheS >> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2059) >> at >> org.apache.ignite.internal.processors.cache.GridCacheAdapter >> $16.call(GridCacheAdapter.java:2057) >> at >> org.apache.ignite.internal.processors.cache.GridCacheContext >> $3.call(GridCacheContext.java:1406) >> at >> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >> (IgniteUtils.java:6621) >> at >> org.apache.ignite.internal.processors.closure.GridClosurePro >> cessor$2.body(GridClosureProcessor.java:954) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >> rker.java:110) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> After this I have to restart the cluster and application or have >> "transactions deadlock" message. >> >> And I checked: network is fine, no OOME in the ignite and cassandra >> (cassandra logs don't say anything about it). I understand that this is >> issue of persistence into cassandra but don't know how can I fix it. And >> after some attempts ignite node is stopped. I tried to change write behind >> parameters and disable it at all - but have the same issue. May be I have >> to >> see in the other place? May be this is important - i use >> IgniteDataStreamer >> for reading values from kafka (earlier I used the simple cache operations >> and didn't have this problem at all). >> >> final IgniteDataStreamer<String, BinaryObject> streamer = >> ignite.dataStreamer(callCache().getName()); >> streamer.autoFlushFrequency(5000); >> streamer.keepBinary(true); >> streamer.perNodeBufferSize(5120); >> >> >> And one more question about persistence settings. If I'm working with >> BinaryObjects and ignite doesn't know about my class types, should I use >> valuePersistence class = "org.apache.ignite.binary.BinaryObject"? I did >> it >> because peer class loading doesn't work for my own classes and I don't >> want >> to add new jars into all ignite nodes and all my apps. >> >> Thanks for help! >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> >> >> -- >> >> Best regards, >> Andrey V. Mashenkov >> > >
