[
https://issues.apache.org/jira/browse/IGNITE-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582331#comment-15582331
]
Ivan Veselovsky commented on IGNITE-4037:
-----------------------------------------
The main point there is that total amount of memory we have (heap, off-heap,
mapped, whatever.) is limited, and is not very large. So, if we use mapped
memory, we can only use a small (limited size) memory window(s) that we can
move over a huge file(s). This window re-positioning is expensive operation,
and this is why stream access is much efficient on files than random I/O.
If we use HadoopSkipList (or any other sorted collection as well), when we
mutate it (e.g. insert a new elements into the sorted collection), we need to
re-position the memory window randomly to find the insertion point and write
the new k-V pair. This leads to random access on disk (in case of mapped memory
many window re-positionings), and drops the performance.
The alternative is to perform sorting (random access) in a small (fixed size)
memory buffer, then spill it to disk, and then merge the sorted sequences
together using the merge sort algorithm step. This uses only stream-like disk
access, and, afaik, this is what both Hadoop and Spark do. Currently I don't
see better alternative to that, so my suggestion is to use similar
implementation in Ignite.
As a memory buffer for sorting HadoopSkipList can be used. The only change I
would suggest is to use org.apache.hadoop.io.RawComparator#compare because it
allows to avoid full deserialization for keys comparison, though, still
requires to get an on-heap byte array. For that reason may be an on-heap
collection is even more preferable.
> High memory consumption when executing TeraSort Hadoop example
> --------------------------------------------------------------
>
> Key: IGNITE-4037
> URL: https://issues.apache.org/jira/browse/IGNITE-4037
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 1.6
> Reporter: Ivan Veselovsky
> Assignee: Ivan Veselovsky
> Fix For: 1.7
>
>
> When executing TeraSort Hadoop example, we observe high memory consumption
> that frequently leads to cluster malfunction.
> The problem can be reproduced in unit test, even with 1 node, and with not
> huge input data set as 100Mb.
> Dump analysis shows that memory is taken in various queues:
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService#queue
>
> and
> task queue of
> org.apache.ignite.internal.processors.hadoop.jobtracker.HadoopJobTracker#evtProcSvc
> .
> Since objects stored in these queues hold byte arrays of significant size,
> memory if consumed very fast.
> It looks like real cause of the problem is that some tasks are blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
