Ivan Veselovsky commented on IGNITE-4037:

The main point there is that total amount of memory we have (heap, off-heap, 
mapped, whatever.) is limited, and is not very large. So, if we use mapped 
memory, we can only use a small (limited size) memory window(s) that we can 
move over a huge file(s). This window re-positioning is expensive operation, 
and this is why stream access is much efficient on files than random I/O. 
If we use HadoopSkipList (or any other sorted collection as well), when we 
mutate it (e.g. insert a new elements into the sorted collection), we need to 
re-position the memory window randomly to find the insertion point and write 
the new k-V pair. This leads to random access on disk (in case of mapped memory 
many window re-positionings), and drops the performance. 
The alternative is to perform sorting (random access) in a small (fixed size) 
memory buffer, then spill it to disk, and then merge the sorted sequences 
together using the merge sort algorithm step. This uses only stream-like disk 
access, and, afaik, this is what both Hadoop and Spark do. Currently I don't 
see better alternative to that, so my suggestion is to use similar 
implementation in Ignite. 

As a memory buffer for sorting HadoopSkipList can be used. The only change I 
would suggest is to use org.apache.hadoop.io.RawComparator#compare because it 
allows to avoid full deserialization for keys comparison, though, still 
requires to get an on-heap byte array. For that reason may be an on-heap 
collection is even more preferable.

> High memory consumption when executing TeraSort Hadoop example
> --------------------------------------------------------------
>                 Key: IGNITE-4037
>                 URL: https://issues.apache.org/jira/browse/IGNITE-4037
>             Project: Ignite
>          Issue Type: Bug
>    Affects Versions: 1.6
>            Reporter: Ivan Veselovsky
>            Assignee: Ivan Veselovsky
>             Fix For: 1.7
> When executing TeraSort Hadoop example, we observe high memory consumption 
> that frequently leads to cluster malfunction.
> The problem can be reproduced in unit test, even with 1 node, and with not 
> huge input data set as 100Mb. 
> Dump analysis shows that  memory is taken in various queues: 
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService#queue
> and 
> task queue of 
> org.apache.ignite.internal.processors.hadoop.jobtracker.HadoopJobTracker#evtProcSvc
>   .
> Since objects stored in these queues hold byte arrays of significant size, 
> memory if consumed very fast.
> It looks like real cause of the problem is that some tasks are blocked.

This message was sent by Atlassian JIRA

Reply via email to