Improve data loading speed from persistant data storage

I am loading data from Oracle database using *cache.loadCache()* command, but it is taking more time in data loading(Takes about *5 minutes to load 300 MB* of data). Can we use *Ignitedatastreamer* here to improve data loading speed? or is there any other way for bulk loading of data from

Re: Ignite cluster

Hi Val, I agree with you on "separate clusters running". but ignite is stopped. Separate clusters has no meaning. agree? Thanks On 22 December 2016 at 00:25, vkulichenko wrote: > Anil, > > All this is very use case specific. STOP is used by default because it's

Re: LoadCache Performance decreases with the size of the cache

Steve, Can you show your indexing configuration? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/LoadCache-Performance-decreases-with-the-size-of-the-cache-tp9645p9688.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Off heap memory contents in linux

Ignite does not track the size of each entry. However, you can use CacheMetrics.getOffHeapAllocatedSize() and CacheMetrics.getOffHeapEntriesCount() to get an approximation. -Val -- View this message in context:

Re: Variation in BinaryObjectImpl object size with number of entries

Hi, Generally, having large heaps is not a very good practice, because eventually it will likely cause long GC pauses and performance degradations. I would recommend to scale out by adding more nodes, or use off-heap memory to store your data. [1]

Re: What are typical load time for millions of records

Hi, I am using the partition capability with multiple threads to improve the data loading, but referring to my other post : I see that the performance are not constant with respect to the size of the cache. Regards -- View this message in context:

Re: LoadCache Performance decreases with the size of the cache

Hi Val and thanks for the reply. I narrowed it down a little bit. The problem comes from the indexing. I tried with no indexing/query fields and then half the fields queryable and it turns out that with indexing the loading performances decrease over time. First, I would like to know whether it

Re: Variation in BinaryObjectImpl object size with number of entries

JVM version I have used is 1.8. So for heap size > 32 GB object size of BinaryObjectImpl will always be 56 Bytes? Also, calculations done at link http://apacheignite.gridgain.org/v1.8/docs/capacity-planning , are they for cache size

Ignite questions

Hi, I started investigating Apache Ignite as a proof-of-concept for production cluster-wide key-value store and distributed counters. I run several tests using REST API on a 3 node cluster; 1 backup; cacheMode Partitioned; synchronization mode FULL_SYNC. Questions: 1. When the Ignite instance is

Re: Variation in BinaryObjectImpl object size with number of entries

When heap size exceeds 32Gb reference compression is disabled, now JVM uses 64-bit references. What's your version of JVM? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Variation-in-BinaryObjectImpl-object-size-with-number-of-entries-tp9673p9678.html Sent from

Off heap memory contents in linux

I want to see per entry memory consumption in case of off heap memory in ignite. Is there any tool by which I can see memory consumption of only off heap entries in ignite(excluding JVM heap usage)? -- View this message in context:

Cache.invoke are terribly slow and can't update cache

Hi experts, I try to update cache by delta by calling invoke. This is the code, it is terribly slow and run out of Memory and no cache updated. Please help. table.getRows().forEach(r -> cache.invoke(r.getKey(), (entry, args) -> { BinaryObject bo = entry.getValue(); if

Re: Variation in BinaryObjectImpl object size with number of entries

Following arguments are passed when I load 39M entries -server -Xms35g -Xmx35g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:NewSize=128m -XX:MaxNewSize=128m -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=1024 -XX:+UseCMSInitiatingOccupancyOnly

Re: Variation in BinaryObjectImpl object size with number of entries

What arguments do you pass to JVM when load 39M entries? What is the heap size in that case? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Variation-in-BinaryObjectImpl-object-size-with-number-of-entries-tp9673p9674.html Sent from the Apache Ignite Users

Variation in BinaryObjectImpl object size with number of entries

For number of entries = 150, BinaryObjectImpl object size is 40 Bytes for both key and value whereas, For number of entries = 3900, BinaryObjectImpl object size is 56 Bytes for both key and value