On Mon, Jan 09, 2012 at 01:29:39PM +0800, James wrote:
> I am build the solr index on the hadoop, and at reduce step I run the task
> that merge the indexes, each part of index is about 1G, I have 10 indexes to
> merge them together, I always get the java heap memory exhausted, the heap
> size is about 2G also. I wonder which part use these so many memory. And how
> to avoid the OOM during the merge process.
There are three issues in there. You should first try to find out which
one it is (it's not clear to me based on your question):
- Java heap memory: you can set that as a start option of the JVM.
You set the maximum with the -Xmxn start option. You get an
OutOfMemory exception if you reach that (no idea wheter the
SOLR code bubbles this up, but there are experts on that here).
- Operating system limit: you can set the limit for a process's
use of resources (memory, among others). Typically, Linux based
systems are shipped with unlimited memory setting; Ralf already
posted how to check/set that.
The situation here is a bit complicated, because there are
different limits (memory size vs. virtual memory size, mainly)
and they are exercised differently depending on the allocation
pattern. Anyway, I'd expect malloc() returning NULL in this
case and the Java runtime translating it (again) into an OutOfMemory
exception.
- Now the OOM killer is quite another kettle of fish. AFAIK, it's
Linux-specific. Once the global system memory is more-or-less
exhausted, the kernel kills some applications to try to improve
the situation. There's some heuristic in deciding which application
to kill, and there are some knobs to help the kernel in this
decision. I'd recommend [1]; after reading *that* you know all :-)
You know you've run into that by looking at the system log.
[1] <https://lwn.net/Articles/317814/>
--
Tomás Zerolo
Axel Springer AG
Axel Springer media Systems
BILD Produktionssysteme
Axel-Springer-Straße 65
10888 Berlin
Tel.: +49 (30) 2591-72875
[email protected]
www.axelspringer.de
Axel Springer AG, Sitz Berlin, Amtsgericht Charlottenburg, HRB 4998
Vorsitzender des Aufsichtsrats: Dr. Giuseppe Vita
Vorstand: Dr. Mathias Döpfner (Vorsitzender)
Jan Bayer, Ralph Büchi, Lothar Lanz, Dr. Andreas Wiele
