Re: 回复： Ignite Data Streamer Hung after a period

[Ilya Kasnacheev]

Hello!

The reason for slow index building, as well as workaround, is described in:
http://apache-ignite-users.70518.x6.nabble.com/Performance-degradation-in-case-of-high-volumes-tp27150p27204.html

Regards,
-- 
Ilya Kasnacheev


чт, 28 февр. 2019 г. в 14:34, Justin Ji :

> I have tried to load data without indexes, but it does not have any help!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 回复： Ignite Data Streamer Hung after a period

[Justin Ji]

I have tried to load data without indexes, but it does not have any help!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 回复： Ignite Data Streamer Hung after a period

[Justin Ji]

I have tried to load data without indexes, but it does not have any help!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 回复： Ignite Data Streamer Hung after a period

[Justin Ji]

After I dig in the issue, I found that the streamer threads are waiting for
building the index.
This looks normal in a database based system, the more data, the slower
insertion.
But ignite is a wide used system, I think other people may encounter this
problem, and have ways to improve the performance. 

Would appreciate anyone who can give me some advices.

"data-streamer-stripe-2-#11%nx-s-ignite-001%" #30 prio=5 os_prio=0
tid=0x5571cbdba800 nid=0x95 waiting on condition [0x7f8e8c9ed000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
   at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
   at
org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:145)
   at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.read(AsyncFileIO.java:95)
   at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:351)
   at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:328)
   at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:312)
   at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779)
   at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624)
   at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
   at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
   at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
   at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:149)
   at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
   at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
   at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:167)
   at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:46)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4482)
   at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:209)
   at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:46)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4469)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4389)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:83)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:278)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4816)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4801)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158)
   at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2336)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2348)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2086)
   at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2066)
   at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:247)
   at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466)
   at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:659)
   at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866)
   at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403)
   at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1393)
   at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1257)
   at

?????? Ignite Data Streamer Hung after a period

[BinaryTree]

fheapManagerImpl.java:1257)
   at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1529)
   at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:352)
   at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3602)
   at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2774)
   at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:2125)
   at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
   at 
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
   at 
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
   at 
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
   at 
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
   at 
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
   at 
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
   at 
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
   at 
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
   at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
   at java.lang.Thread.run(Thread.java:748)







--  --
??: "BinaryTree";
: 2019??2??28??(??) 11:27
??: 
"user";"ilya.kasnacheev";

: ?? Ignite Data Streamer Hung after a period



Ilya 


I attached the thread dump logs. I have three ignite nodes, every node dumped 
four thread files.
Looking forward to your reply.




--  --
??: "BinaryTree";
: 2019??2??28??(??) 10:28
??????: 
"user";"ilya.kasnacheev";

: ?? Ignite Data Streamer Hung after a period



Thank for your reply. 
1. Yes, I have persistence.
2. I think the cache store is not the bottleneck, because the skipStore is 
enabled when loading data.
IgniteDataStreamer streamer = 
ignite.dataStreamer(IgniteCacheKey.DATA_POINT_NEW.getCode());
streamer.skipStore(true);
streamer.keepBinary(true);
streamer.perNodeBufferSize(1);
streamer.perNodeParallelOperations(32);





--  --
??: "Ilya Kasnacheev";
????????: 2019??2??27??(??) 9:59
??: "user";

: Re: Ignite Data Streamer Hung after a period



Hello!



It's hard to say. Do you have persistence? Are you sure that cache store is not 
the bottleneck?


I would start with gathering thread dumps from whole cluster when in stuck 
state.


Regards,

-- 

Ilya Kasnacheev









, 27 . 2019 ??. ?? 15:06, Justin Ji :

Dmitry  - 
 
 I also encountered this problem.
 
 I used both persistence and indexing, when I loaded 20 million records, the
 loading speed became much slower than before, but the CPU of the ignite
 server is low.
 
 
<http://apache-ignite-users.70518.x6.nabble.com/file/t2000/WX20190227-200059.png>
 
 
 Here is my cache configuration:
 
 CacheConfiguration cacheCfg = new CacheConfiguration();
 cacheCfg.setName(cacheName);
 cacheCfg.setCacheMode(CacheMode.PARTITIONED);
 cacheCfg.setBackups(1);
 cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
 
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
 cacheCfg.setWriteThrough(true);
 cacheCfg.setWriteBehindEnabled(true);
 cacheCfg.setWriteBehindFlushThreadCount(2);
 cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
 cacheCfg.setWriteBehindFlushSize(409600);
 cacheCfg.setWriteBehindBatchSize(1024);
 cacheCfg.setStoreKeepBinary(true);
 cacheCfg.setQueryParallelism(16);
 cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
 cacheCfg.setRebalanceThrottle(100);
 CacheKeyConfiguration cacheKeyConfiguration = new
 CacheKeyConfiguration(DpKey.class);
 cacheCfg.setKeyConfiguration(cacheKeyConfiguration);
 
 List entities = Lists.newArrayList();
 
 QueryEntity entity = new QueryEntity(DpKey.class.getName(),
 DpCache.class.getName());
 entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());
 
 LinkedHashMap map = new LinkedHashMap<>();
 map.put("id", "java.lang.String");
 map.put("gmtCreate", "java.lang.Long");
 map.put("gmtModified", "java.lang.Long");

?????? Ignite Data Streamer Hung after a period

[BinaryTree]

Thank for your reply. 
1. Yes, I have persistence.
2. I think the cache store is not the bottleneck, because the skipStore is 
enabled when loading data.
IgniteDataStreamer streamer = 
ignite.dataStreamer(IgniteCacheKey.DATA_POINT_NEW.getCode());
streamer.skipStore(true);
streamer.keepBinary(true);
streamer.perNodeBufferSize(1);
streamer.perNodeParallelOperations(32);





--  --
??: "Ilya Kasnacheev";
: 2019??2??27??(??) 9:59
??: "user";

????: Re: Ignite Data Streamer Hung after a period



Hello!



It's hard to say. Do you have persistence? Are you sure that cache store is not 
the bottleneck?


I would start with gathering thread dumps from whole cluster when in stuck 
state.


Regards,

-- 

Ilya Kasnacheev









, 27 . 2019 ??. ?? 15:06, Justin Ji :

Dmitry  - 
 
 I also encountered this problem.
 
 I used both persistence and indexing, when I loaded 20 million records, the
 loading speed became much slower than before, but the CPU of the ignite
 server is low.
 
 
<http://apache-ignite-users.70518.x6.nabble.com/file/t2000/WX20190227-200059.png>
 
 
 Here is my cache configuration:
 
 CacheConfiguration cacheCfg = new CacheConfiguration();
 cacheCfg.setName(cacheName);
 cacheCfg.setCacheMode(CacheMode.PARTITIONED);
 cacheCfg.setBackups(1);
 cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
 
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
 cacheCfg.setWriteThrough(true);
 cacheCfg.setWriteBehindEnabled(true);
 cacheCfg.setWriteBehindFlushThreadCount(2);
 cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
 cacheCfg.setWriteBehindFlushSize(409600);
 cacheCfg.setWriteBehindBatchSize(1024);
 cacheCfg.setStoreKeepBinary(true);
 cacheCfg.setQueryParallelism(16);
 cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
 cacheCfg.setRebalanceThrottle(100);
 CacheKeyConfiguration cacheKeyConfiguration = new
 CacheKeyConfiguration(DpKey.class);
 cacheCfg.setKeyConfiguration(cacheKeyConfiguration);
 
 List entities = Lists.newArrayList();
 
 QueryEntity entity = new QueryEntity(DpKey.class.getName(),
 DpCache.class.getName());
 entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());
 
 LinkedHashMap map = new LinkedHashMap<>();
 map.put("id", "java.lang.String");
 map.put("gmtCreate", "java.lang.Long");
 map.put("gmtModified", "java.lang.Long");
 map.put("devId", "java.lang.String");
 map.put("dpId", "java.lang.Integer");
 map.put("code", "java.lang.String");
 map.put("name", "java.lang.String");
 map.put("customName", "java.lang.String");
 map.put("mode", "java.lang.String");
 map.put("type", "java.lang.String");
 map.put("value", "java.lang.String");
 map.put("rawValue", byte[].class.getName());
 map.put("time", "java.lang.Long");
 map.put("status", "java.lang.Boolean");
 map.put("uuid", "java.lang.String");
 
 entity.setFields(map);
 QueryIndex devIdIdx = new QueryIndex("devId");
 devIdIdx.setName("idx_devId");
 devIdIdx.setInlineSize(128);
 List indexes = Lists.newArrayList(devIdIdx);
 entity.setIndexes(indexes);
 
 entities.add(entity);
 cacheCfg.setQueryEntities(entities);
 
 
 Can you give me some advice on where to start solving these problems?
 
 
 
 
 --
 Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Data Streamer Hung after a period

[Ilya Kasnacheev]

Hello!

It's hard to say. Do you have persistence? Are you sure that cache store is
not the bottleneck?

I would start with gathering thread dumps from whole cluster when in stuck
state.

Regards,
-- 
Ilya Kasnacheev


ср, 27 февр. 2019 г. в 15:06, Justin Ji :

> Dmitry  -
>
> I also encountered this problem.
>
> I used both persistence and indexing, when I loaded 20 million records, the
> loading speed became much slower than before, but the CPU of the ignite
> server is low.
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2000/WX20190227-200059.png>
>
>
> Here is my cache configuration:
>
> CacheConfiguration cacheCfg = new CacheConfiguration();
> cacheCfg.setName(cacheName);
> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> cacheCfg.setBackups(1);
> cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>
> cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
> cacheCfg.setWriteThrough(true);
> cacheCfg.setWriteBehindEnabled(true);
> cacheCfg.setWriteBehindFlushThreadCount(2);
> cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
> cacheCfg.setWriteBehindFlushSize(409600);
> cacheCfg.setWriteBehindBatchSize(1024);
> cacheCfg.setStoreKeepBinary(true);
> cacheCfg.setQueryParallelism(16);
> cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
> cacheCfg.setRebalanceThrottle(100);
> CacheKeyConfiguration cacheKeyConfiguration = new
> CacheKeyConfiguration(DpKey.class);
> cacheCfg.setKeyConfiguration(cacheKeyConfiguration);
>
> List entities = Lists.newArrayList();
>
> QueryEntity entity = new QueryEntity(DpKey.class.getName(),
> DpCache.class.getName());
> entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());
>
> LinkedHashMap map = new LinkedHashMap<>();
> map.put("id", "java.lang.String");
> map.put("gmtCreate", "java.lang.Long");
> map.put("gmtModified", "java.lang.Long");
> map.put("devId", "java.lang.String");
> map.put("dpId", "java.lang.Integer");
> map.put("code", "java.lang.String");
> map.put("name", "java.lang.String");
> map.put("customName", "java.lang.String");
> map.put("mode", "java.lang.String");
> map.put("type", "java.lang.String");
> map.put("value", "java.lang.String");
> map.put("rawValue", byte[].class.getName());
> map.put("time", "java.lang.Long");
> map.put("status", "java.lang.Boolean");
> map.put("uuid", "java.lang.String");
>
> entity.setFields(map);
> QueryIndex devIdIdx = new QueryIndex("devId");
> devIdIdx.setName("idx_devId");
> devIdIdx.setInlineSize(128);
> List indexes = Lists.newArrayList(devIdIdx);
> entity.setIndexes(indexes);
>
> entities.add(entity);
> cacheCfg.setQueryEntities(entities);
>
>
> Can you give me some advice on where to start solving these problems?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite Data Streamer Hung after a period

[Justin Ji]

Dmitry  - 

I also encountered this problem.

I used both persistence and indexing, when I loaded 20 million records, the
loading speed became much slower than before, but the CPU of the ignite
server is low.


 

Here is my cache configuration:

CacheConfiguration cacheCfg = new CacheConfiguration();
cacheCfg.setName(cacheName);
cacheCfg.setCacheMode(CacheMode.PARTITIONED);
cacheCfg.setBackups(1);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
cacheCfg.setWriteThrough(true);
cacheCfg.setWriteBehindEnabled(true);
cacheCfg.setWriteBehindFlushThreadCount(2);
cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
cacheCfg.setWriteBehindFlushSize(409600);
cacheCfg.setWriteBehindBatchSize(1024);
cacheCfg.setStoreKeepBinary(true);
cacheCfg.setQueryParallelism(16);
cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);
cacheCfg.setRebalanceThrottle(100);
CacheKeyConfiguration cacheKeyConfiguration = new
CacheKeyConfiguration(DpKey.class);
cacheCfg.setKeyConfiguration(cacheKeyConfiguration);

List entities = Lists.newArrayList();

QueryEntity entity = new QueryEntity(DpKey.class.getName(),
DpCache.class.getName());
entity.setTableName(IgniteTableKey.T_DATA_POINT_NEW.getCode());

LinkedHashMap map = new LinkedHashMap<>();
map.put("id", "java.lang.String");
map.put("gmtCreate", "java.lang.Long");
map.put("gmtModified", "java.lang.Long");
map.put("devId", "java.lang.String");
map.put("dpId", "java.lang.Integer");
map.put("code", "java.lang.String");
map.put("name", "java.lang.String");
map.put("customName", "java.lang.String");
map.put("mode", "java.lang.String");
map.put("type", "java.lang.String");
map.put("value", "java.lang.String");
map.put("rawValue", byte[].class.getName());
map.put("time", "java.lang.Long");
map.put("status", "java.lang.Boolean");
map.put("uuid", "java.lang.String");

entity.setFields(map);
QueryIndex devIdIdx = new QueryIndex("devId");
devIdIdx.setName("idx_devId");
devIdIdx.setInlineSize(128);
List indexes = Lists.newArrayList(devIdIdx);
entity.setIndexes(indexes);

entities.add(entity);
cacheCfg.setQueryEntities(entities);


Can you give me some advice on where to start solving these problems?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Data Streamer Hung after a period

[dkarachentsev]

Hi,

Blocked threads show only the fact that there are no tasks to process in
pool. Do you use persistence and/or indexing? Could you please attach your
configs and logs from all nodes? Please take few sequential thread dumps
when throughput is low.

Thanks!
-Dmitry 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/