Re: Question about Ignite persistence disk space used after clearing cache

[Denis Magda]

Hi Scott,

The disk space is not compacted even after you clear the entire cache. The
compaction feature will be introduced to Ignite soon. So, the metric shows
the allocated size. This doc section suggests an approach for the actual
size calculation:
https://www.gridgain.com/docs/latest/administrators-guide/monitoring-metrics/metrics#allocated-space-vs-actual-size-of-data



-
Denis


On Fri, Sep 25, 2020 at 1:52 PM Scott Prater  wrote:

> I have a question about how the off-heap usage is reported when Ignite
> persistence is configured.  I have a single node set up.  I stored about
> 1GB of items in the cache, then cleared the cache (remotely, using the Java
> thin client:  ClientCache.clear()).
>
> I then verified that the items were no longer in the cache.
>
> However, when I look at the Ignite log, I do not see that the disk space
> was freed:
>
> [2020-09-25T11:17:36,299][INFO ][grid-timeout-worker-#23][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=db4ed295, uptime=00:34:00.176]
> ^-- H/N/C [hosts=1, nodes=1, CPUs=8]
> ^-- CPU [cur=0.2%, avg=0.3%, GC=0%]
> ^-- PageMemory [pages=250315]
> ^-- Heap [used=180MB, free=94.85%, comm=438MB]
> ^-- Off-heap [used=989MB, free=88.35%, comm=8392MB]
> ^--   sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^--   default_region region [used=989MB, free=87.92%, comm=8192MB]
> ^--   metastoreMemPlc region [used=0MB, free=99.94%, comm=0MB]
> ^--   TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=998MB]
> ^--   sysMemPlc region [used=0MB]
> ^--   default_region region [used=998MB]
> ^--   metastoreMemPlc region [used=0MB]
> ^--   TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
>
> "Ignite persistence [used=998MB]" seems to indicate that 1GB of data is
> still in the cache.  Is this simply a report of the disk space *allocated*,
> or is actual disk space in use?  Is there a way to get both measurements?
>
> thanks,
>
> -- Scott
>

Re: Fast Load Large Datasets

[Denis Magda]

Vladimir,

It was a pleasure working with you on this article. Igniters, you'll find
useful tips & tricks on how to load large tables. Read and bookmark.

-
Denis


On Thu, Sep 24, 2020 at 9:43 PM  wrote:

>
> Igniters,
>
> My tutorial post about loading big tables into Apache Ignite has finally
> arrived [1]. Many thanks to @Denis Magda and @Ksenia Romanova for their
> valuable help.
>
> [1]
> https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api
>
> Vladimir
>

Question about Ignite persistence disk space used after clearing cache

[Scott Prater]

I have a question about how the off-heap usage is reported when Ignite
persistence is configured.  I have a single node set up.  I stored about
1GB of items in the cache, then cleared the cache (remotely, using the Java
thin client:  ClientCache.clear()).

I then verified that the items were no longer in the cache.

However, when I look at the Ignite log, I do not see that the disk space
was freed:

[2020-09-25T11:17:36,299][INFO ][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=db4ed295, uptime=00:34:00.176]
^-- H/N/C [hosts=1, nodes=1, CPUs=8]
^-- CPU [cur=0.2%, avg=0.3%, GC=0%]
^-- PageMemory [pages=250315]
^-- Heap [used=180MB, free=94.85%, comm=438MB]
^-- Off-heap [used=989MB, free=88.35%, comm=8392MB]
^--   sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^--   default_region region [used=989MB, free=87.92%, comm=8192MB]
^--   metastoreMemPlc region [used=0MB, free=99.94%, comm=0MB]
^--   TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=998MB]
^--   sysMemPlc region [used=0MB]
^--   default_region region [used=998MB]
^--   metastoreMemPlc region [used=0MB]
^--   TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]

"Ignite persistence [used=998MB]" seems to indicate that 1GB of data is
still in the cache.  Is this simply a report of the disk space *allocated*,
or is actual disk space in use?  Is there a way to get both measurements?

thanks,

-- Scott

Re: OutOfMemoryException with Persistence and Eviction Enabled

[Andrei Aleksandrov]

Hi,

It looks like that I found the issue:

https://issues.apache.org/jira/browse/IGNITE-8917

When you use *put *or *removeAll *in persistence cache with more data 
than data region size throw IgniteOutOfMemoryException. Data streamer 
looks like don't affected by this ticket.


The WA is pretty simple - use bigger data region.

BR,
Andrei

9/24/2020 11:44 PM, Mitchell Rathbun (BLOOMBERG/ 731 LEX) пишет:
I tried doubling it from 200 MB to 400 MB. My initial test worked, but 
as I increased the number of fields that I was writing per entry, the 
same issue occurred again. So it seems to just increase the capacity 
of what can be written, not actually prevent the exception from 
occurring. I guess my main question is why is it possible for Ignite 
to get an OOME when persistence and eviction are enabled? It seems 
like if there are a lot of writes, performance should degrade as the 
in memory cache evicts members of the cache, but no exceptions should 
occur. The error is always "Failed to find a page for eviction", which 
doesn't really make sense when eviction is enabled. What are the 
internal structures that Ignite holds in Off-heap memory?
Also, why isn't this an issue when using IgniteDataStreamer if the 
issue has to do with space for internal structures? Wouldn't Ignite 
need the same internal structures for either case?

From: user@ignite.apache.org At: 09/24/20 11:42:21
To: user@ignite.apache.org  Subject: 
Re: OutOfMemoryException with Persistence and Eviction Enabled


Hi, Did you try to increase the DataRegion size a little bit? It
looks like 190 MB isn't enough for some internal structures that
Ignite stores in OFF-HEAP except the data. I suggest you increase
the data region size to for example 512 MB - 1024 MB and take a
look at how it will work. If you still will see the issue then I
guess we should create the ticket: 1)Collect the logs 2)Provide
the java code example 3)Provide the configuration of the nodes
After that, we can take a look more deeply into it and if it's an
issue then file JIRA. BR, Andrei

9/23/2020 7:36 PM, Mitchell Rathbun (BLOOMBERG/ 731 LEX) пишет:

Here is the exception:
Sep 22, 2020 7:58:22 PM java.util.logging.LogManager$RootLogger log
SEVERE: Critical system error detected. Will be handled
accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.mem.IgniteOutOfMemoryException: Out of memory in data
region [name=customformulacalcrts, initSize=190.7 MiB,
maxSize=190.7 MiB, persistenceEnabled=true] Try the following:
^-- Increase maximum off-heap memory size
(DataRegionConfiguration.maxSize)
^-- Enable Ignite persistence
(DataRegionConfiguration.persistenceEnabled)
^-- Enable eviction or expiration policies]]
class org.apache.ignite.internal.mem.IgniteOutOfMemoryException:
Out of memory in data region [name=customformulacalcrts,
initSize=190.7 MiB, maxSize=190.7 MiB, persistenceEnabled=true]
Try the following:
^-- Increase maximum off-heap memory size
(DataRegionConfiguration.maxSize)
^-- Enable Ignite persistence
(DataRegionConfiguration.persistenceEnabled)
^-- Enable eviction or expiration policies
at

org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.allocatePage(PageMemoryImpl.java:607)
at

org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.allocateDataPage(AbstractFreeList.java:464)
at

org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:491)
at

org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.insertDataRow(CacheFreeListImpl.java:59)
at

org.apache.ignite.internal.processors.cache.persistence.freelist.CacheFreeListImpl.insertDataRow(CacheFreeListImpl.java:35)
at

org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:103)
at

org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1691)
at

org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.createRow(GridCacheOffheapManager.java:1910)
at

org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5701)
at

org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:5643)
at

org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3719)
at