I'm seeing our integration tests hang. The cause is something in
Random2LruPageEvictionTracker, i'm able to reproduce this 100% of time on
many different laptops and in our ci/cd. I was able to work around this by
disabling PageEviction in our Default Data Region during the tests. I don't
see this in our production clusters.
Does anyone have an idea of how i can fix it without disabling eviction?
I'm using Ignite 2.8.0.
One thing to note is that I'm not sure what to make of the line
Random2LruPageEvictionTracker.evictDataPage(Random2LruPageEvictionTracker.java:152),
but it seemed to simply loop forever when stepping through it in my IDE.
Stack Trace:
"Test worker" #22 prio=5 os_prio=31 cpu=299703.41ms elapsed=317.18s
tid=0x00007ff3cfc8c800 nid=0x7203 runnable [0x0000700005b38000]
java.lang.Thread.State: RUNNABLE at
org.apache.ignite.internal.processors.cache.persistence.evict.Random2LruPageEvictionTracker.evictDataPage(Random2LruPageEvictionTracker.java:152)
at
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.ensureFreeSpace(IgniteCacheDatabaseSharedManager.java:1086)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.ensureFreeSpace(GridCacheMapEntry.java:4513)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1461)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.localFinish(GridNearTxLocal.java:3850)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.doFinish(GridNearTxFinishFuture.java:440)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.finish(GridNearTxFinishFuture.java:390)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:4129)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$25.apply(GridNearTxLocal.java:4118)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:354)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:4118)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commit(GridNearTxLocal.java:4086)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:587)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:556)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.retryTopologySafe(DataStructuresProcessor.java:1664)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:556)
at
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.reentrantLock(DataStructuresProcessor.java:1361)
at
org.apache.ignite.internal.IgniteKernal.reentrantLock(IgniteKernal.java:4136)
at jdk.internal.reflect.GeneratedMethodAccessor713.invoke(Unknown Source)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
at com.example.ignite.IgniteInitializer$1.invoke(IgniteInitializer.java:158)
at com.sun.proxy.$Proxy205.reentrantLock(Unknown Source) at
com.example.data.store.jdbc.cache.CacheService.getCount(CacheService.java:47)
at
com.example.data.store.jdbc.cache.CacheService$$FastClassBySpringCGLIB$$7efa9131.invoke()......
Configuration: I tried different permutations, the only thing that worked
was disabling eviction. I also tried LRU instead of LRU_2 but that didn't
work either.
@Bean public IgniteConfiguration igniteConfiguration(
@Value("#{'${IGNITE_HOME:}' ?: systemProperties['java.io.tmpdir']}") String
igniteHome, SymphonyIgniteProperties
symphonyIgniteProperties, Optional
dataStorageConfiguration, TcpDiscoverySpi tcpDiscoverySpi) {
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setIgniteInstanceName(symphonyIgniteProperties.getIgniteInstanceName());
dataStorageConfiguration.ifPresent(dss ->
cfg.setDataStorageConfiguration(dss)); cfg.setIgniteHome(igniteHome);
cfg.setPeerClassLoadingEnabled(true); cfg.setIncludeEventTypes(new
int[] {
EVT_CACHE_STARTED });
System.setProperty(IgniteSystemProperties.IGNITE_QUIET, "true");
System.setProperty(IgniteSystemProperties.IGNITE_UPDATE_NOTIFIER, "false");
cfg.setGridLogger(new Slf4jLogger());
cfg.setDiscoverySpi(tcpDiscoverySpi);
cfg.setCommunicationSpi(new StandaloneNoopCommunicationSpi());
cfg.setClientMode(false); cfg.setDataStreamerThreadPoolSize(2);
cfg.setPublicThreadPoolSize(2); cfg.setSystemThreadPoolSize(2);
cfg.setStripedPoolSize(2); cfg.setIgfsThreadPoolSize(2);
cfg.setManagementThreadPoolSize(2);
cfg.setServiceThreadPoolSize(2);
cfg.setQueryThreadPoolSize(2);
cfg.setPeerClassLoadingThreadPoolSize(2);
// setting ConnectorConfiguration to null removes these thread pools:
//
"grid-nio-worker-tcp-rest-x", "nio-acceptor-tcp-rest-#18%%" // and
"session-timeout-worker-#13%%" cfg.setConnectorConfiguration(null);
cfg.setClientConnectorConfiguration(null);
cfg.setRebalanceThreadPoolSize(1); return cfg; }
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
