Hi, On Tue, Nov 19, 2013 at 1:57 AM, Chetan Mehrotra <chetan.mehro...@gmail.com> wrote: > On some systems running on Oak (on Windows) where lots of nodes get > created we are seeing high RAM usage. The Java process memory (shown > directly under task manager. See note on Working Set below) remains > within expected range but system starts paging memory quite a bit and > slows down. > > On checking the runtime state following things can be observed > * FileStore has 186 TarFile and each TarFile refers to 256MB > MappedByteBuffer taking around 46 GB of memory > * The JVM process constantly shows 50% CPU usage. Checking further > indictes that AsynchUpdate is running and Lucene indexing is being > performed > * Windows Task Manager shows 7 .66 GB (total 8 GB) overall memory > usage with Java process only showing 1.4GB usage. No other process > account for so high usage > * Checking the Working Set [1] shows 7 GB memory being used by Java > process alone > > From my basic understanding of the Memory Mapped file usage it should > not cause msuch resource crunch and OS would be able to manage the > memory. However probably in Oak case all these TarFiles are getting > accessed frequently which leads to these memory mapped pages held in > physical memory.
Hmm, I wonder if we do have a problem here or if the system is just working as designed. If you have 8GB of memory on the system and no other processes using much of it, then it would be natural and expected for the operating system to use most of the available memory for caching the memory mapped files that are actively being accessed by TarMK. The 50% CPU usage you mention suggests that the system isn't actually being blocked by the paging behavior. Instead it's just busy crunching through the content, which causes newly accessed data to be paged in and old to be discarded. The same disk IO would be required without memory mapping. If thrashing was occurring, you'd expect to see much lower CPU usage. > Should we put upper cap on number of TarFiles opened? If this indeed is a problem, a better solution would probably be to limit the maximum working set size at the operating system level. Alternatively we could switch to using normal file IO for bulk segments (ones that only contain parts of large binaries). > If any other data needs to be collected to determine the problem cause > further then let me know? Can you reduce the behavior to a standalone test case? Or you might want to try running the system with a 32 bit JVM, which should force TarMK to use normal file IO with a fixed-size buffer instead of memory mapping. [1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms686237(v=vs.85).aspx BR, Jukka Zitting
