Hi Tobias, I created a ticket to fix the issue: https://issues.apache.org/jira/browse/IGNITE-6853
2017-10-25 16:23 GMT+03:00 Tobias Eriksson <tobias.eriks...@qvantel.com>: > Hi Andrey et al > > I believe I now know what the problem is, the Cassandra session is > refreshed, but before it is a prepared statement is created/used and there, > and so using a new session with an old prepared statement is not working. > > > > The way to reproduce is > > 1) Start Ignite Server Node > > 2) Start client which inserts a batch of 100 elements > > 3) End client > > 4) Now Ignite Server Node returns the Cassandra Session to the pool > > 5) Wait 5+ minutes > > 6) Now Ignite Server Node has does a clean-up of the “unused” > Cassandra sessions > > 7) Start client which inserts a batch of 100 elements > > 8) Boom ! The exception starts to happen > > > > Reason is > > 1) Execute is called for a BATCH > > 2) Prepared-statement is reused since there is a cache of those > > 3) It is about to do session().execute( batch ) > > 4) BUT the call to session() results in refreshing the session, and > this is where the prepared statements as the old session new them are > cleaned up > > 5) Now it is looping over 100 times with a NEW session but with an > OLD prepared statement > > > > This is a bug, > > > > -Tobias > > > > > > *From: *Andrey Mashenkov <andrey.mashen...@gmail.com> > *Reply-To: *"user@ignite.apache.org" <user@ignite.apache.org> > *Date: *Wednesday, 25 October 2017 at 14:12 > *To: *"user@ignite.apache.org" <user@ignite.apache.org> > *Subject: *Re: Ignite-cassandra module issue > > > > Hi Tobias, > > > > What ignite version do you use? May be this was already fixed in latest > one? > > I see related fix inclueded in upcoming 2.3 version. > > > > See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end > logic, so session should be closed in proper way. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5897 > > > > > > On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson < > tobias.eriks...@qvantel.com> wrote: > > Hi > Sorry did not include the context when I replied > Has anyone been able to resolve this problem, cause I have it too on and > off > In fact it sometimes happens just like that, e.g. I have been running my > Ignite client and then stop it, and then it takes a while and run it again, > and all by a sudden this error shows up. An that is the first thing that > happens, and there is NOT a massive amount of load on Cassandra at that > time. But I have also seen it when I hammer Ignite/Cassandra with > updates/inserts. > > This is a deal-breaker for me, I need to understand how to fix this, cause > having this in production is not an option. > > -Tobias > > > Hi! > I'm using the cassandra as persistence store for my caches and have one > issue by handling a huge data (via IgniteDataStreamer from kafka). > Ignite Configuration: > final IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); > igniteConfiguration.setIgniteInstanceName("test"); > igniteConfiguration.setClientMode(true); > igniteConfiguration.setGridLogger(new Slf4jLogger()); > igniteConfiguration.setMetricsLogFrequency(0); > igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi()); > final BinaryConfiguration binaryConfiguration = new BinaryConfiguration(); > binaryConfiguration.setCompactFooter(false); > igniteConfiguration.setBinaryConfiguration(binaryConfiguration); > igniteConfiguration.setPeerClassLoadingEnabled(true); > final MemoryPolicyConfiguration memoryPolicyConfiguration = new > MemoryPolicyConfiguration(); > memoryPolicyConfiguration.setName("3Gb_Region_Eviction"); > memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L); > memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L); > > memoryPolicyConfiguration.setPageEvictionMode( > DataPageEvictionMode.RANDOM_2_LRU); > final MemoryConfiguration memoryConfiguration = new MemoryConfiguration(); > memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration); > igniteConfiguration.setMemoryConfiguration(memoryConfiguration); > > Cache configuration: > final CacheConfiguration<String, BinaryObject> cacheConfiguration = new > CacheConfiguration<>(); > cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC); > cacheConfiguration.setStoreKeepBinary(true); > cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); > cacheConfiguration.setBackups(0); > cacheConfiguration.setStatisticsEnabled(false); > cacheConfiguration.setName("TestCache"); > > cacheConfiguration.setReadThrough(true); > cacheConfiguration.setWriteThrough(true); > > cacheConfiguration.setWriteBehindEnabled(true); > cacheConfiguration.setWriteBehindFlushFrequency(10000); > cacheConfiguration.setWriteBehindFlushSize(0); > cacheConfiguration.setWriteBehindFlushThreadCount(2); > cacheConfiguration.setWriteBehindBatchSize(10000); > > > final CassandraCacheStoreFactory<String, BinaryObject> > cacheStoreFactory = new CassandraCacheStoreFactory<>(); > final DataSource dataSource = new DataSource(); > dataSource.setContactPoints(contactPoints); > dataSource.setReadConsistency("ONE"); > dataSource.setWriteConsistency("ONE"); > dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new > RoundRobinPolicy())); > cacheStoreFactory.setDataSource(dataSource); > > final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\" > table=\"%s\">" + > "<keyPersistence class=\"%s\" strategy=\"%s\"/>" + > "<valuePersistence class=\"%s\" strategy=\"%s\"/>" + > "</persistence>"; > final KeyValuePersistenceSettings settings = new > KeyValuePersistenceSettings( > String.format(CASSANDRA_PERSISTENCE, "test", "test_table", > "java.lang.String", "PRIMITIVE", > "org.apache.ignite.binary.BinaryObject", "BLOB")); > cacheStoreFactory.setPersistenceSettings(settings); > cacheConfiguration.setCacheStoreFactory(cacheStoreFactory); > > When application works some time (hour or more, may be less - from time to > time) I see this exceptions on the ignite nodes: > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072] > ^-- H/N/C [hosts=3, nodes=3, CPUs=96] > ^-- CPU [cur=0%, avg=1.86%, GC=0%] > ^-- PageMemory [pages=118064] > ^-- Heap [used=4800MB, free=53.12%, comm=10240MB] > ^-- Non heap [used=78MB, free=-1%, comm=80MB] > ^-- Public thread pool [active=0, idle=32, qSize=0] > ^-- System thread pool [active=0, idle=32, qSize=0] > ^-- Outbound messages queue [size=0] > [15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList > [name=null, buckets=256, dataPages=102080, reusePages=0] > [15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared > statement cluster error detected, refreshing Cassandra session > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra > session refreshed > [15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for > 332ms > [15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for > 332ms > [15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared > statement cluster error detected, refreshing Cassandra session > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed > [15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed > [15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed > [15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed > [15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed > [15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed > [15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed > [15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 > attempt > to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed > [15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed > [15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2 > attempt to execute Cassandra CQL statement: select "value" from > "triggers"."callback" where "key"=?; > [15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra > session refreshed > [15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for > 229ms > [15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared > statement cluster error detected, another thread already fixed the problem > com.datastax.driver.core.exceptions.InvalidQueryException: Tried to > execute > unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have > used a PreparedStatement that was created with another Cluster instance. > at > com.datastax.driver.core.SessionManager.makeRequestMessage( > SessionManager.java:568) > at > com.datastax.driver.core.SessionManager.executeAsync( > SessionManager.java:131) > at > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) > at > org.apache.ignite.cache.store.cassandra.session. > CassandraSessionImpl.execute(CassandraSessionImpl.java:144) > at > org.apache.ignite.cache.store.cassandra.CassandraCacheStore. > load(CassandraCacheStore.java:168) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) > at > org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper. > load(CacheStoreBalancingWrapper.java:98) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter. > java:316) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAllFromStore( > GridCacheStoreManagerAdapter.java:416) > at > org.apache.ignite.internal.processors.cache.store. > GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter. > java:382) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2059) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call( > GridCacheAdapter.java:2057) > at > org.apache.ignite.internal.processors.cache.GridCacheContext$3.call( > GridCacheContext.java:1406) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils. > java:6621) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body( > GridClosureProcessor.java:954) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > After this I have to restart the cluster and application or have > "transactions deadlock" message. > > And I checked: network is fine, no OOME in the ignite and cassandra > (cassandra logs don't say anything about it). I understand that this is > issue of persistence into cassandra but don't know how can I fix it. And > after some attempts ignite node is stopped. I tried to change write behind > parameters and disable it at all - but have the same issue. May be I have > to > see in the other place? May be this is important - i use IgniteDataStreamer > for reading values from kafka (earlier I used the simple cache operations > and didn't have this problem at all). > > final IgniteDataStreamer<String, BinaryObject> streamer = > ignite.dataStreamer(callCache().getName()); > streamer.autoFlushFrequency(5000); > streamer.keepBinary(true); > streamer.perNodeBufferSize(5120); > > > And one more question about persistence settings. If I'm working with > BinaryObjects and ignite doesn't know about my class types, should I use > valuePersistence class = "org.apache.ignite.binary.BinaryObject"? I did > it > because peer class loading doesn't work for my own classes and I don't want > to add new jars into all ignite nodes and all my apps. > > Thanks for help! > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > > > > > > -- > > Best regards, > Andrey V. Mashenkov >
