Re: Ignite.NET very slow Cache Loading

[Saifullah Zahid]

Hi Pavel,

Thanks for  your reply.
There are about 40 fileds TransactionsDetail and size is about 400 bytes.
In Taskbar I can see the process is taking about 23GB
I used 20GB heap-size.

Also I have noticed that c: drive have only 3GB empty space.

Thanks,
Saif

On Thu, Mar 16, 2017 at 11:16 AM, Pavel Tupitsyn 
wrote:

> I thought that you use DataStreamer to load data into cache for some
> reason.
> As I now see in gitter, you actually use ICacheStore.LoadCache.
>
> So my next thought is that slowdown can be caused by GC because of low
> memory.
> How much memory one row takes? Can you show TransactionsDetail code, does
> it have a lot of fields?
> Can you attach to your process with jvisualvm.exe (part of Java JDK) and
> see how much heap memory is actually used?
> Another option is to look at Private Working Set in Task Manager.
>
> Please also attach Ignite log.
>
> On Wed, Mar 15, 2017 at 8:13 PM, Saifullah Zahid 
> wrote:
>
>> I tried that but it is throwing exception
>>
>> class org.apache.ignite.IgniteCheckedException: Cannot enable
>> write-behind (writer or store is not provided) for cache:
>>
>> If I comment following settings
>>
>> ReadThrough = true,
>> WriteThrough = true,
>> WriteBehindEnabled = true,
>>
>> cache gets created but without loading data from database.
>>
>> thanks,
>>
>>
>>
>> On Wed, Mar 15, 2017 at 10:00 PM, Pavel Tupitsyn 
>> wrote:
>>
>>> Why use Spring?
>>> I propose to do just one thing - comment out the CacheStoreFactory line.
>>>
>>> On Wed, Mar 15, 2017 at 7:30 PM, Saifullah Zahid 
>>> wrote:
>>>
 Hi,

 If I disable CacheStoreFactor then I have to provide the
 CacheConfiguration in spring xml
 But I don't know how to provide Database and table mapping in spring
 xml, I tried following configuration but it is failing.
 I am not sure if it is right way to do?

 
 http://www.springframework.org/schema/beans;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xmlns:util="http://www.springframework.org/schema/util;
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.or
 g/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.or
 g/schema/util/spring-util.xsd">

 
 
 >>> value="com.microsoft.sqlserver.jdbc.SQLServerDriver"
 />
 
 
 
 
  
  
 
  
 
 
 
 
 
 
 
 >>> value="transactionsdetail" />
 
 
 
 

 Getting following error

  Failed to instantiate configuration from Spring XML

 Thanks,
 Saif

 On Wed, Mar 15, 2017 at 8:29 PM, Pavel Tupitsyn 
 wrote:

> Have you tried commenting out cache store as I descrived above?
>
> On Wed, Mar 15, 2017 at 6:10 PM, Saifullah Zahid 
> wrote:
>
>> Hi,
>>
>> following is my load cache method
>> if I comments out following line of code, it does not slows down
>> act(GetKeyValue(item), item);
>>
>> public void LoadCache(Action act, params object[]
>> args)
>> {
>> using (DbConnection _connection = Create())
>> {
>> using (DbCommand command = _connection.CreateCommand())
>> {
>> MakeLoadCacheCommand(command, args);
>> using (var reader = command.ExecuteReader())
>> {
>> while (reader.Read())
>> {
>> var item = new T();
>> Map(reader, item); //populate object
>> * act(GetKeyValue(item), item); //insert into cache*
>> }
>> }
>> }
>> }
>> }
>>
>> Looks like issue is when inserting data into cache.
>>
>> Thanks,
>> Saif
>>
>>
>> On Wed, Mar 15, 2017 at 7:02 PM, Pavel Tupitsyn > > wrote:
>>
>>> Hi,
>>>
>>> You have cache store configured, which probably causes the slowdown.
>>> Please try to disable cache store (remove CacheStoreFactory from
>>> config) and see if it makes any difference.
>>>
>>> On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid <
>>> saif.m...@gmail.com> wrote:
>>>
 Hi,

 I am facing an issue on cache loading, In the start cache loads
 quickly but after some time it becomes very slow almost 1 row per 
 second.
 There are about 4 million rows in a table.
 OS = Windows Server 2012
 RAM = 64 GB
 Node Heap config is 16 GB.
 Following is Cache configuration

 TransactionsDetailStore = m_cache.GetOrCreateCache(new CacheConfiguration("transactionssdetail",
 

Re: Ignite.NET very slow Cache Loading

[Saifullah Zahid]

I tried that but it is throwing exception

class org.apache.ignite.IgniteCheckedException: Cannot enable write-behind
(writer or store is not provided) for cache:

If I comment following settings

ReadThrough = true,
WriteThrough = true,
WriteBehindEnabled = true,

cache gets created but without loading data from database.

thanks,



On Wed, Mar 15, 2017 at 10:00 PM, Pavel Tupitsyn 
wrote:

> Why use Spring?
> I propose to do just one thing - comment out the CacheStoreFactory line.
>
> On Wed, Mar 15, 2017 at 7:30 PM, Saifullah Zahid 
> wrote:
>
>> Hi,
>>
>> If I disable CacheStoreFactor then I have to provide the
>> CacheConfiguration in spring xml
>> But I don't know how to provide Database and table mapping in spring xml,
>> I tried following configuration but it is failing.
>> I am not sure if it is right way to do?
>>
>> 
>> http://www.springframework.org/schema/beans;
>>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>xmlns:util="http://www.springframework.org/schema/util;
>>xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.or
>> g/schema/beans/spring-beans.xsd
>>http://www.springframework.org/schema/util
>>http://www.springframework.or
>> g/schema/util/spring-util.xsd">
>>
>> 
>> 
>> > value="com.microsoft.sqlserver.jdbc.SQLServerDriver"
>> />
>> 
>> 
>> 
>> 
>>  
>>  
>> 
>>  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> > value="transactionsdetail" />
>> 
>> 
>> 
>> 
>>
>> Getting following error
>>
>>  Failed to instantiate configuration from Spring XML
>>
>> Thanks,
>> Saif
>>
>> On Wed, Mar 15, 2017 at 8:29 PM, Pavel Tupitsyn 
>> wrote:
>>
>>> Have you tried commenting out cache store as I descrived above?
>>>
>>> On Wed, Mar 15, 2017 at 6:10 PM, Saifullah Zahid 
>>> wrote:
>>>
 Hi,

 following is my load cache method
 if I comments out following line of code, it does not slows down
 act(GetKeyValue(item), item);

 public void LoadCache(Action act, params object[] args)
 {
 using (DbConnection _connection = Create())
 {
 using (DbCommand command = _connection.CreateCommand())
 {
 MakeLoadCacheCommand(command, args);
 using (var reader = command.ExecuteReader())
 {
 while (reader.Read())
 {
 var item = new T();
 Map(reader, item); //populate object
 * act(GetKeyValue(item), item); //insert into cache*
 }
 }
 }
 }
 }

 Looks like issue is when inserting data into cache.

 Thanks,
 Saif


 On Wed, Mar 15, 2017 at 7:02 PM, Pavel Tupitsyn 
 wrote:

> Hi,
>
> You have cache store configured, which probably causes the slowdown.
> Please try to disable cache store (remove CacheStoreFactory from
> config) and see if it makes any difference.
>
> On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid 
> wrote:
>
>> Hi,
>>
>> I am facing an issue on cache loading, In the start cache loads
>> quickly but after some time it becomes very slow almost 1 row per second.
>> There are about 4 million rows in a table.
>> OS = Windows Server 2012
>> RAM = 64 GB
>> Node Heap config is 16 GB.
>> Following is Cache configuration
>>
>> TransactionsDetailStore = m_cache.GetOrCreateCache> TransactionsDetail>(new CacheConfiguration("transactionssdetail",
>> typeof(TransactionsDetail))
>> {
>> CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
>> true),
>> ReadThrough = true,
>> WriteThrough = true,
>> WriteBehindEnabled = true,
>> KeepBinaryInStore = false,
>> WriteBehindFlushThreadCount = 4,
>> WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
>> MemoryMode = CacheMemoryMode.OffheapTiered,
>> OffHeapMaxMemory = 0,
>> EvictionPolicy = new LruEvictionPolicy { MaxSize = 100 },
>> WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
>> });
>>
>> Am I missing some configuration?
>> Kindly see if anyone have idea?
>>
>> Thanks,
>> Saif
>>
>
>

>>>
>>
>

Re: Ignite.NET very slow Cache Loading

[Pavel Tupitsyn]

Why use Spring?
I propose to do just one thing - comment out the CacheStoreFactory line.

On Wed, Mar 15, 2017 at 7:30 PM, Saifullah Zahid 
wrote:

> Hi,
>
> If I disable CacheStoreFactor then I have to provide the
> CacheConfiguration in spring xml
> But I don't know how to provide Database and table mapping in spring xml,
> I tried following configuration but it is failing.
> I am not sure if it is right way to do?
>
> 
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xmlns:util="http://www.springframework.org/schema/util;
>xsi:schemaLocation="http://www.springframework.org/schema/beans
>http://www.springframework.
> org/schema/beans/spring-beans.xsd
>http://www.springframework.org/schema/util
>http://www.springframework.
> org/schema/util/spring-util.xsd">
>
> 
> 
>  value="com.microsoft.sqlserver.jdbc.SQLServerDriver"
> />
>  value="jdbc:sqlserver://localhost:1433;databaseName=TransactionsDB"
> />
> 
> 
> 
>  
>  
> 
>  
> 
> 
> 
> 
> 
> 
> 
>  value="transactionsdetail" />
> 
> 
> 
> 
>
> Getting following error
>
>  Failed to instantiate configuration from Spring XML
>
> Thanks,
> Saif
>
> On Wed, Mar 15, 2017 at 8:29 PM, Pavel Tupitsyn 
> wrote:
>
>> Have you tried commenting out cache store as I descrived above?
>>
>> On Wed, Mar 15, 2017 at 6:10 PM, Saifullah Zahid 
>> wrote:
>>
>>> Hi,
>>>
>>> following is my load cache method
>>> if I comments out following line of code, it does not slows down
>>> act(GetKeyValue(item), item);
>>>
>>> public void LoadCache(Action act, params object[] args)
>>> {
>>> using (DbConnection _connection = Create())
>>> {
>>> using (DbCommand command = _connection.CreateCommand())
>>> {
>>> MakeLoadCacheCommand(command, args);
>>> using (var reader = command.ExecuteReader())
>>> {
>>> while (reader.Read())
>>> {
>>> var item = new T();
>>> Map(reader, item); //populate object
>>> * act(GetKeyValue(item), item); //insert into cache*
>>> }
>>> }
>>> }
>>> }
>>> }
>>>
>>> Looks like issue is when inserting data into cache.
>>>
>>> Thanks,
>>> Saif
>>>
>>>
>>> On Wed, Mar 15, 2017 at 7:02 PM, Pavel Tupitsyn 
>>> wrote:
>>>
 Hi,

 You have cache store configured, which probably causes the slowdown.
 Please try to disable cache store (remove CacheStoreFactory from
 config) and see if it makes any difference.

 On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid 
 wrote:

> Hi,
>
> I am facing an issue on cache loading, In the start cache loads
> quickly but after some time it becomes very slow almost 1 row per second.
> There are about 4 million rows in a table.
> OS = Windows Server 2012
> RAM = 64 GB
> Node Heap config is 16 GB.
> Following is Cache configuration
>
> TransactionsDetailStore = m_cache.GetOrCreateCache TransactionsDetail>(new CacheConfiguration("transactionssdetail",
> typeof(TransactionsDetail))
> {
> CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
> true),
> ReadThrough = true,
> WriteThrough = true,
> WriteBehindEnabled = true,
> KeepBinaryInStore = false,
> WriteBehindFlushThreadCount = 4,
> WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
> MemoryMode = CacheMemoryMode.OffheapTiered,
> OffHeapMaxMemory = 0,
> EvictionPolicy = new LruEvictionPolicy { MaxSize = 100 },
> WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
> });
>
> Am I missing some configuration?
> Kindly see if anyone have idea?
>
> Thanks,
> Saif
>


>>>
>>
>

Re: Ignite.NET very slow Cache Loading

[Saifullah Zahid]

Hi,

If I disable CacheStoreFactor then I have to provide the CacheConfiguration
in spring xml
But I don't know how to provide Database and table mapping in spring xml, I
tried following configuration but it is failing.
I am not sure if it is right way to do?


http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xmlns:util="http://www.springframework.org/schema/util;
   xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans.xsd
   http://www.springframework.org/schema/util

http://www.springframework.org/schema/util/spring-util.xsd;>








 
 

 













Getting following error

 Failed to instantiate configuration from Spring XML

Thanks,
Saif

On Wed, Mar 15, 2017 at 8:29 PM, Pavel Tupitsyn 
wrote:

> Have you tried commenting out cache store as I descrived above?
>
> On Wed, Mar 15, 2017 at 6:10 PM, Saifullah Zahid 
> wrote:
>
>> Hi,
>>
>> following is my load cache method
>> if I comments out following line of code, it does not slows down
>> act(GetKeyValue(item), item);
>>
>> public void LoadCache(Action act, params object[] args)
>> {
>> using (DbConnection _connection = Create())
>> {
>> using (DbCommand command = _connection.CreateCommand())
>> {
>> MakeLoadCacheCommand(command, args);
>> using (var reader = command.ExecuteReader())
>> {
>> while (reader.Read())
>> {
>> var item = new T();
>> Map(reader, item); //populate object
>> * act(GetKeyValue(item), item); //insert into cache*
>> }
>> }
>> }
>> }
>> }
>>
>> Looks like issue is when inserting data into cache.
>>
>> Thanks,
>> Saif
>>
>>
>> On Wed, Mar 15, 2017 at 7:02 PM, Pavel Tupitsyn 
>> wrote:
>>
>>> Hi,
>>>
>>> You have cache store configured, which probably causes the slowdown.
>>> Please try to disable cache store (remove CacheStoreFactory from
>>> config) and see if it makes any difference.
>>>
>>> On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid 
>>> wrote:
>>>
 Hi,

 I am facing an issue on cache loading, In the start cache loads quickly
 but after some time it becomes very slow almost 1 row per second.
 There are about 4 million rows in a table.
 OS = Windows Server 2012
 RAM = 64 GB
 Node Heap config is 16 GB.
 Following is Cache configuration

 TransactionsDetailStore = m_cache.GetOrCreateCache(new CacheConfiguration("transactionssdetail",
 typeof(TransactionsDetail))
 {
 CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
 true),
 ReadThrough = true,
 WriteThrough = true,
 WriteBehindEnabled = true,
 KeepBinaryInStore = false,
 WriteBehindFlushThreadCount = 4,
 WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
 MemoryMode = CacheMemoryMode.OffheapTiered,
 OffHeapMaxMemory = 0,
 EvictionPolicy = new LruEvictionPolicy { MaxSize = 100 },
 WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
 });

 Am I missing some configuration?
 Kindly see if anyone have idea?

 Thanks,
 Saif

>>>
>>>
>>
>

Re: Ignite.NET very slow Cache Loading

[Pavel Tupitsyn]

Have you tried commenting out cache store as I descrived above?

On Wed, Mar 15, 2017 at 6:10 PM, Saifullah Zahid 
wrote:

> Hi,
>
> following is my load cache method
> if I comments out following line of code, it does not slows down
> act(GetKeyValue(item), item);
>
> public void LoadCache(Action act, params object[] args)
> {
> using (DbConnection _connection = Create())
> {
> using (DbCommand command = _connection.CreateCommand())
> {
> MakeLoadCacheCommand(command, args);
> using (var reader = command.ExecuteReader())
> {
> while (reader.Read())
> {
> var item = new T();
> Map(reader, item); //populate object
> * act(GetKeyValue(item), item); //insert into cache*
> }
> }
> }
> }
> }
>
> Looks like issue is when inserting data into cache.
>
> Thanks,
> Saif
>
>
> On Wed, Mar 15, 2017 at 7:02 PM, Pavel Tupitsyn 
> wrote:
>
>> Hi,
>>
>> You have cache store configured, which probably causes the slowdown.
>> Please try to disable cache store (remove CacheStoreFactory from config)
>> and see if it makes any difference.
>>
>> On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid 
>> wrote:
>>
>>> Hi,
>>>
>>> I am facing an issue on cache loading, In the start cache loads quickly
>>> but after some time it becomes very slow almost 1 row per second.
>>> There are about 4 million rows in a table.
>>> OS = Windows Server 2012
>>> RAM = 64 GB
>>> Node Heap config is 16 GB.
>>> Following is Cache configuration
>>>
>>> TransactionsDetailStore = m_cache.GetOrCreateCache>> TransactionsDetail>(new CacheConfiguration("transactionssdetail",
>>> typeof(TransactionsDetail))
>>> {
>>> CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
>>> true),
>>> ReadThrough = true,
>>> WriteThrough = true,
>>> WriteBehindEnabled = true,
>>> KeepBinaryInStore = false,
>>> WriteBehindFlushThreadCount = 4,
>>> WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
>>> MemoryMode = CacheMemoryMode.OffheapTiered,
>>> OffHeapMaxMemory = 0,
>>> EvictionPolicy = new LruEvictionPolicy { MaxSize = 100 },
>>> WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
>>> });
>>>
>>> Am I missing some configuration?
>>> Kindly see if anyone have idea?
>>>
>>> Thanks,
>>> Saif
>>>
>>
>>
>

Re: Ignite.NET very slow Cache Loading

[Saifullah Zahid]

Hi,

following is my load cache method
if I comments out following line of code, it does not slows down
act(GetKeyValue(item), item);

public void LoadCache(Action act, params object[] args)
{
using (DbConnection _connection = Create())
{
using (DbCommand command = _connection.CreateCommand())
{
MakeLoadCacheCommand(command, args);
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
var item = new T();
Map(reader, item); //populate object
* act(GetKeyValue(item), item); //insert into cache*
}
}
}
}
}

Looks like issue is when inserting data into cache.

Thanks,
Saif


On Wed, Mar 15, 2017 at 7:02 PM, Pavel Tupitsyn 
wrote:

> Hi,
>
> You have cache store configured, which probably causes the slowdown.
> Please try to disable cache store (remove CacheStoreFactory from config)
> and see if it makes any difference.
>
> On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid 
> wrote:
>
>> Hi,
>>
>> I am facing an issue on cache loading, In the start cache loads quickly
>> but after some time it becomes very slow almost 1 row per second.
>> There are about 4 million rows in a table.
>> OS = Windows Server 2012
>> RAM = 64 GB
>> Node Heap config is 16 GB.
>> Following is Cache configuration
>>
>> TransactionsDetailStore = m_cache.GetOrCreateCache> TransactionsDetail>(new CacheConfiguration("transactionssdetail",
>> typeof(TransactionsDetail))
>> {
>> CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
>> true),
>> ReadThrough = true,
>> WriteThrough = true,
>> WriteBehindEnabled = true,
>> KeepBinaryInStore = false,
>> WriteBehindFlushThreadCount = 4,
>> WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
>> MemoryMode = CacheMemoryMode.OffheapTiered,
>> OffHeapMaxMemory = 0,
>> EvictionPolicy = new LruEvictionPolicy { MaxSize = 100 },
>> WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
>> });
>>
>> Am I missing some configuration?
>> Kindly see if anyone have idea?
>>
>> Thanks,
>> Saif
>>
>
>

Re: Ignite.NET very slow Cache Loading

[Pavel Tupitsyn]

Hi,

You have cache store configured, which probably causes the slowdown.
Please try to disable cache store (remove CacheStoreFactory from config)
and see if it makes any difference.

On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid 
wrote:

> Hi,
>
> I am facing an issue on cache loading, In the start cache loads quickly
> but after some time it becomes very slow almost 1 row per second.
> There are about 4 million rows in a table.
> OS = Windows Server 2012
> RAM = 64 GB
> Node Heap config is 16 GB.
> Following is Cache configuration
>
> TransactionsDetailStore = m_cache.GetOrCreateCache TransactionsDetail>(new CacheConfiguration("transactionssdetail",
> typeof(TransactionsDetail))
> {
> CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
> true),
> ReadThrough = true,
> WriteThrough = true,
> WriteBehindEnabled = true,
> KeepBinaryInStore = false,
> WriteBehindFlushThreadCount = 4,
> WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
> MemoryMode = CacheMemoryMode.OffheapTiered,
> OffHeapMaxMemory = 0,
> EvictionPolicy = new LruEvictionPolicy { MaxSize = 100 },
> WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
> });
>
> Am I missing some configuration?
> Kindly see if anyone have idea?
>
> Thanks,
> Saif
>

Ignite.NET very slow Cache Loading

[Saifullah Zahid]

Hi,

I am facing an issue on cache loading, In the start cache loads quickly but
after some time it becomes very slow almost 1 row per second.
There are about 4 million rows in a table.
OS = Windows Server 2012
RAM = 64 GB
Node Heap config is 16 GB.
Following is Cache configuration

TransactionsDetailStore = m_cache.GetOrCreateCache(new CacheConfiguration("transactionssdetail",
typeof(TransactionsDetail))
{
CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
true),
ReadThrough = true,
WriteThrough = true,
WriteBehindEnabled = true,
KeepBinaryInStore = false,
WriteBehindFlushThreadCount = 4,
WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
MemoryMode = CacheMemoryMode.OffheapTiered,
OffHeapMaxMemory = 0,
EvictionPolicy = new LruEvictionPolicy { MaxSize = 100 },
WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
});

Am I missing some configuration?
Kindly see if anyone have idea?

Thanks,
Saif