After I dig in the issue, I found that the streamer threads are waiting for building the index. This looks normal in a database based system, the more data, the slower insertion. But ignite is a wide used system, I think other people may encounter this problem, and have ways to improve the performance.
Would appreciate anyone who can give me some advices. "data-streamer-stripe-2-#11%nx-s-ignite-001%" #30 prio=5 os_prio=0 tid=0x00005571cbdba800 nid=0x95 waiting on condition [0x00007f8e8c9ed000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177) at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145) at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.read(AsyncFileIO.java:95) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:351) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:328) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:312) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:149) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:167) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:46) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4482) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:209) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:46) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4469) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4389) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:83) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:278) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4816) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4801) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2336) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2086) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2066) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:247) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:659) at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1393) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1257) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1529) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:352) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3602) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2774) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:2125) at org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - <0x00000007767a5b60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "data-streamer-stripe-1-#10%nx-s-ignite-001%" #29 prio=5 os_prio=0 tid=0x00005571cbdb8800 nid=0x94 waiting on condition [0x00007f8e8caee000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177) at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145) at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.read(AsyncFileIO.java:95) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:351) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:328) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:312) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:142) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2326) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2086) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2066) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:247) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:548) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:480) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:659) at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1393) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1257) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1529) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:352) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3602) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2774) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:2125) at org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511) at java.lang.Thread.run(Thread.java:748) ------------------ ???????? ------------------ ??????: "BinaryTree"<[email protected]>; ????????: 2019??2??28??(??????) ????11:27 ??????: "user"<[email protected]>;"ilya.kasnacheev"<[email protected]>; ????: ?????? Ignite Data Streamer Hung after a period Ilya I attached the thread dump logs. I have three ignite nodes, every node dumped four thread files. Looking forward to your reply. ------------------ ???????? ------------------ ??????: "BinaryTree"<[email protected]>; ????????: 2019??2??28??(??????) ????10:28 ??????: "user"<[email protected]>;"ilya.kasnacheev"<[email protected]>; ????: ?????? Ignite Data Streamer Hung after a period Thank for your reply. 1. Yes, I have persistence. 2. I think the cache store is not the bottleneck, because the skipStore is enabled when loading data. IgniteDataStreamer<DpKey, BinaryObject> streamer = ignite.dataStreamer(IgniteCacheKey.DATA_POINT_NEW.getCode()); streamer.skipStore(true); streamer.keepBinary(true); streamer.perNodeBufferSize(10000); streamer.perNodeParallelOperations(32); ------------------ ???????? ------------------ ??????: "Ilya Kasnacheev"<[email protected]>; ????????: 2019??2??27??(??????) ????9:59 ??????: "user"<[email protected]>; ????: Re: Ignite Data Streamer Hung after a period Hello! It's hard to say. Do you have persistence? Are you sure that cache store is not the bottleneck? I would start with gathering thread dumps from whole cluster when in stuck state. Regards, -- Ilya Kasnacheev ????, 27 ????????. 2019 ??. ?? 15:06, Justin Ji <[email protected]>: Dmitry - I also encountered this problem. I used both persistence and indexing, when I loaded 20 million records, the loading speed became much slower than before, but the CPU of the ignite server is low. <http://apache-ignite-users.70518.x6.nabble.com/file/t2000/WX20190227-200059.png> Here is my cache configuration: CacheConfiguration<K, V> cacheCfg = new CacheConfiguration(); cacheCfg.setName(cacheName); cacheCfg.setCacheMode(CacheMode.PARTITIONED); cacheCfg.setBackups(1); cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC); cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class)); cacheCfg.setWriteThrough(true); cacheCfg.setWriteBehindEnabled(true); cacheCfg.setWriteBehindFlushThreadCount(2); cacheCfg.setWriteBehindFlushFrequency(15 * 1000); cacheCfg.setWriteBehindFlushSize(409600); cacheCfg.setWriteBehindBatchSize(1024); cacheCfg.setStoreKeepBinary(true); cacheCfg.setQueryParallelism(16); cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024); cacheCfg.setRebalanceThrottle(100); CacheKeyConfiguration cacheKeyConfiguration = new CacheKeyConfiguration(DpKey.class); cacheCfg.setKeyConfiguration(cacheKeyConfiguration); List<QueryEntity> entities = Lists.newArrayList(); QueryEntity entity = new QueryEntity(DpKey.class.getName(), DpCache.class.getName()); entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode()); LinkedHashMap<String, String> map = new LinkedHashMap<>(); map.put("id", "java.lang.String"); map.put("gmtCreate", "java.lang.Long"); map.put("gmtModified", "java.lang.Long"); map.put("devId", "java.lang.String"); map.put("dpId", "java.lang.Integer"); map.put("code", "java.lang.String"); map.put("name", "java.lang.String"); map.put("customName", "java.lang.String"); map.put("mode", "java.lang.String"); map.put("type", "java.lang.String"); map.put("value", "java.lang.String"); map.put("rawValue", byte[].class.getName()); map.put("time", "java.lang.Long"); map.put("status", "java.lang.Boolean"); map.put("uuid", "java.lang.String"); entity.setFields(map); QueryIndex devIdIdx = new QueryIndex("devId"); devIdIdx.setName("idx_devId"); devIdIdx.setInlineSize(128); List<QueryIndex> indexes = Lists.newArrayList(devIdIdx); entity.setIndexes(indexes); entities.add(entity); cacheCfg.setQueryEntities(entities); Can you give me some advice on where to start solving these problems? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
