Are you getting "GC Overhead limit exceeded" OOME?
I think you could always move IGFS data block cache off heap if it is
not the case already.
I am wondering why you've set block size to 4 GB for Ignite when HDFS
stock configured block size is either 64 MB or 128 MB. Have you tried to
set HDFS block size to 4GB? I am guessing you would get OOME on HDFS
data nodes too.
Kamil
Dňa 2017-04-14 08:50 张帅 napísal(a):
I'm using the latest version of JDK, AKA. 1.8.0_121
The cache is aim to provide a faster read/write performance. But the
availability is more important. 1GB cache is for testing purpose. But
it's the same issue if I write a 1TB file to 64GB cache.
What I mean availability is that Ignite should not exit with OOME.
Slow down write performance is kind of downgrade. If I write directly
to HDFS, I got a write performance of x MB/s. If I write through
Ignite, I got a higher performance y MB/s. It is great if y far more
larger than x, and also acceptable equal to x sometimes, but not
acceptable if HDFS still working but Ignite not working.
Breaking into small blocks is possible because data coming in a kind
of stream. We are always able to pack it whenever we collected 512MB
data.
This issue is not about Cache Eviction Strategy, but about how to
avoid OOME & service not available. Cache eviction would not solve it
because there do have more data than cache capacity.
-----Original Message-----
From: Jörn Franke [mailto:[email protected]]
Sent: Friday, April 14, 2017 2:36 PM
To: [email protected]
Subject: Re: OOM when using Ignite as HDFS Cache
I would not expect any of the things that you mention. A cache is not
supposed to slow down writing. This does not make sense from my point
of view. Splitting a block into several smaller ones is also not
feasible. The data has to go somewhere before splitting.
I think what you refer to is certain cache eviction strategies.
1 GB of cache sounds small for a HDFS cache.
I suggest to enable the default configuration of ignite on HDFS and
then change it step by step to your envisioned configuration.
That being said, a Hadoop platform with a lot of ecosystem components
can be complex, in particular you need to calculate that each of the
components (hive, spark etc) has certain memory assigned or has it
used when jobs are running. So even if you have configured 1 gb
somebody else might have taken it. Less probable but possible is that
your JDK has a bug leading to OOME. You may also try to upgrade it.
On 14. Apr 2017, at 08:12, <[email protected]>
<[email protected]> wrote:
I think it's a kind of misconfiguration. The Ignite document just
mentioned about how to configuration HDFS as a secondary filesystem
but nothing about how to restrict the memory usage to avoid OOME.
https://apacheignite.readme.io/v1.0/docs/igfs-secondary-file-system
Assume I configured the max JVM heap size to 1GB.
1. What would happen if I write very fast before Ignite write data to
HDFS asynchronized?
2. What would happen if I want to write a 2GB file block to Ignite?
I expected:
1. Ignite would slow down the write performance to avoid OOME.
2. Ignite would break the 2GB file block into 512MB blocks & write
them to HDFS to avoid OOME.
Do we have configurations against above behaviors? I dig some items
from source code & Ignite Web Console, but seems they are not working
fine.
<property name="fragmentizerConcurrentFiles" value="3"/> <property
name="dualModeMaxPendingPutsSize" value="10"/> <property
name="blockSize" value="536870912"/> <property name="streamBufferSize"
value="131072"/> <property name="maxSpaceSize" value="6442450944"/>
<property name="maximumTaskRangeLength" value="536870912"/> <property
name="prefetchBlocks" value="2"/> <property
name="sequentialReadsBeforePrefetch" value="5"/> <property
name="defaultMode" value="DUAL_ASYNC" />
I also notice that Ignite write through file block size is set to
64MB. I mean I write a file to Ignite with block size to 4GB, but I
finally found it on HDFS with block size 64MB. Is there any
configuration for it?
-----Original Message-----
From: dkarachentsev [mailto:[email protected]]
Sent: Thursday, April 13, 2017 11:21 PM
To: [email protected]
Subject: Re: OOM when using Ignite as HDFS Cache
Hi Shuai,
Could you please take heap dump on OOME and find what objects consume
memory? There would be a lot of byte[] objects, please find the
nearest GC root for them.
Thanks!
-Dmitry.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/OOM-when-using-Ignite-a
s-HDFS-Cache-tp11900p11956.html Sent from the Apache Ignite Users
mailing list archive at Nabble.com.
