John, To sum up the overloading Mike describes, it looks like you have about 9MM documents taking up 80GB of disk space. That should be fine and won't overload a machine.
However, by adding many indexes (and possibly by having many, many elements per document that are indexed by them?) the docs increase the RAM consumption. Your forest stats shows 12GB or so RAM used by the forests, which could be 10GB+ indexes, so 17GB RAM on the box (including OS, other apps, MarkLogic working memory, etc.) doesn't allow enough room for the indexes. I'm not completely sure if the MAPBIG is due to a single, oversized index in one stand, or an overall lack of room for all the files together, however. Yours, Damon -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Michael Blakeley Sent: Friday, November 04, 2011 11:33 AM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] SVC-MAPBIG: Mapped file too large to map: 5825151760 bytes error At the moment you probably can't do much with the system because of the overloaded memory. But taking a longer view, there may be some possibilities for slimming down the database's memory utilization. Your best option is probably to review your database configuration. Right now you appear to be using about 3-kiB per fragment, which is pretty high. That number is largely driven by index configuration. I'd guess that you have a dozen or more range index values per document? Many applications come in under 512-B per fragment, and it's rare that I see more that 1.5-kiB per fragment. I find it helpful to document the application's use of every full-text index setting and every additional index enabled, so that it's easy to review what the application really needs. By doing this, I have seen applications slim down the average fragment memory by as much as 50%. You could try to eliminate extra fragments. Turning off "automatic directory creation" can help, unless you need it for webdav. Turning off "maintain last modified" can help, unless you are using properties for something important (eg, CPF). Neither of these settings impacts existing fragments, so you would have to arrange for them to be deleted, or clear the database and reload. Most of the fragments saved using this technique will have small memory footprints, on the order of 100-B. You can also get some marginal return by reducing database in-memory limits, which control the sizes of in-memory stands. But there isn't more than about half a GB on the table there, and ingestion performance may suffer. If you make the limits too small, some documents may not ingest. You could try to minimize your group-level caches. That's a somewhat dangerous game, since trimming them too much will result in EXPNTREECACHEFULL and LISTCACHEFULL errors. Those caches generally use about 3/8 of RAM, so this can be a *temporary* way to get a little breathing room while you make other adjustments. But the system is designed to use those caches, and won't perform well if they are too small. -- Mike On 4 Nov 2011, at 08:10 , John Zhong wrote: > Thank you, Mike. And yes, I knew the server was overloaded. I just want to > know if there is another suggestion except upgrading the server. It is > probably no. > > Thanks again. > John > > On Fri, Nov 4, 2011 at 11:00 PM, Michael Blakeley <[email protected]> wrote: > To be blunt: that host is out of memory and over capacity. > > Stop ingestion right away, and revert the configuration changes you have > made. Turning off merges is almost always a mistake: there ought to be a pop > quiz built into the configuration screen. You can easily paint yourself into > a corner. The other changes were almost certainly counterproductive, too. > > How can I tell that you are out of memory? The error message states that > MarkLogic was unable to map a 5-GiB index. That is only one of many indexes > that need to memory-map, so I would guess that the system is badly > overloaded. The rest of the data confirms it: on a 16-GiB host your total > index memory (aka forest memory) is about 12-GiB. You can't use all the host > memory for forest indexes: you'll need about that much again for group-level > caches, and about 25% of RAM for queries, merges, and the OS. For your data > so far, you ought to have at least 32-GB RAM. > > Do you know where you are in your total load? Are you about 10% done, or > closer to 75% done? > > With the system in its current state, it will probably use a lot of virtual > memory for paging, and performance will probably be unacceptable. Am I right > in thinking this is a VM? If so, you may be able to upgrade the VM memory > without much trouble. Virtual or not, if a memory upgrade isn't practical > then I would recommend clearing the database and starting over, and this time > stop before you are out of memory. > > -- Mike > > On 4 Nov 2011, at 00:33 , John Zhong wrote: > > > Hi all, > > > > I am running MarkLogic 4.1-6 version and encountered the below error today: > > > > XDMP-FORESTERR: Error in merge of forest: SVC-MAPBIG: Mapped file too large > > to map: 5825151760 bytes: 'D:\Program > > Files\MarkLogic\Data\Forests\forest1\00001cde\580f5ba44789ea78_dateTime' > > > > The MarkLogic instance is running on a EC2 server has 1 CUP with 2 cores, > > 17.1GB RAM, 64 bit windows server 2008. We have created only one database > > with one forest, now the forest has 6 stands, total is 86GB, and there is > > still 300GB disk space available. > > > > active fragments deleted fragments on disk size > > in memory size > > stand 1: 2,922,206 3,320,621 36,246 MB > > 4,162 MB > > stand 2: 1,111,868 1,795,830 39,987 MB > > 6,836 MB > > stand 3: 63,545 107,616 3,588 MB > > 606 MB > > stand 4: 7,620 24,492 925 > > MB 163 MB > > stand 5: 5,638 6,882 479 > > MB 79 MB > > stand 6: 11,757 28,626 1,761 > > MB 319 MB > > TOTAL: 4,122,634 5,284,067 82,986 MB > > 12,165 MB > > > > We enable the default merge setting and tried to modify the merge max size > > to 2048 then merge manually, but still failed. > > > > Now, we disable the merge. > > > > Any sugestion will be very appreciate!! > > > > Thank you in advance. > > John > > > > > > _______________________________________________ > > General mailing list > > [email protected] > > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
