To be blunt: that host is out of memory and over capacity. Stop ingestion right away, and revert the configuration changes you have made. Turning off merges is almost always a mistake: there ought to be a pop quiz built into the configuration screen. You can easily paint yourself into a corner. The other changes were almost certainly counterproductive, too.
How can I tell that you are out of memory? The error message states that MarkLogic was unable to map a 5-GiB index. That is only one of many indexes that need to memory-map, so I would guess that the system is badly overloaded. The rest of the data confirms it: on a 16-GiB host your total index memory (aka forest memory) is about 12-GiB. You can't use all the host memory for forest indexes: you'll need about that much again for group-level caches, and about 25% of RAM for queries, merges, and the OS. For your data so far, you ought to have at least 32-GB RAM. Do you know where you are in your total load? Are you about 10% done, or closer to 75% done? With the system in its current state, it will probably use a lot of virtual memory for paging, and performance will probably be unacceptable. Am I right in thinking this is a VM? If so, you may be able to upgrade the VM memory without much trouble. Virtual or not, if a memory upgrade isn't practical then I would recommend clearing the database and starting over, and this time stop before you are out of memory. -- Mike On 4 Nov 2011, at 00:33 , John Zhong wrote: > Hi all, > > I am running MarkLogic 4.1-6 version and encountered the below error today: > > XDMP-FORESTERR: Error in merge of forest: SVC-MAPBIG: Mapped file too large > to map: 5825151760 bytes: 'D:\Program > Files\MarkLogic\Data\Forests\forest1\00001cde\580f5ba44789ea78_dateTime' > > The MarkLogic instance is running on a EC2 server has 1 CUP with 2 cores, > 17.1GB RAM, 64 bit windows server 2008. We have created only one database > with one forest, now the forest has 6 stands, total is 86GB, and there is > still 300GB disk space available. > > active fragments deleted fragments on disk size > in memory size > stand 1: 2,922,206 3,320,621 36,246 MB > 4,162 MB > stand 2: 1,111,868 1,795,830 39,987 MB > 6,836 MB > stand 3: 63,545 107,616 3,588 MB > 606 MB > stand 4: 7,620 24,492 925 MB > 163 MB > stand 5: 5,638 6,882 479 > MB 79 MB > stand 6: 11,757 28,626 1,761 MB > 319 MB > TOTAL: 4,122,634 5,284,067 82,986 MB > 12,165 MB > > We enable the default merge setting and tried to modify the merge max size to > 2048 then merge manually, but still failed. > > Now, we disable the merge. > > Any sugestion will be very appreciate!! > > Thank you in advance. > John > > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
