Mike, Thank you again for your detail suggestions. I did try few options as you mentioned, such as reducing db in memory size. But as you said, yes, I enabled many index configuration, so the only choice is to upgrade the server.
John On Fri, Nov 4, 2011 at 11:33 PM, Michael Blakeley <[email protected]> wrote: > At the moment you probably can't do much with the system because of the > overloaded memory. But taking a longer view, there may be some > possibilities for slimming down the database's memory utilization. > > Your best option is probably to review your database configuration. Right > now you appear to be using about 3-kiB per fragment, which is pretty high. > That number is largely driven by index configuration. I'd guess that you > have a dozen or more range index values per document? Many applications > come in under 512-B per fragment, and it's rare that I see more that > 1.5-kiB per fragment. > > I find it helpful to document the application's use of every full-text > index setting and every additional index enabled, so that it's easy to > review what the application really needs. By doing this, I have seen > applications slim down the average fragment memory by as much as 50%. > > You could try to eliminate extra fragments. Turning off "automatic > directory creation" can help, unless you need it for webdav. Turning off > "maintain last modified" can help, unless you are using properties for > something important (eg, CPF). Neither of these settings impacts existing > fragments, so you would have to arrange for them to be deleted, or clear > the database and reload. Most of the fragments saved using this technique > will have small memory footprints, on the order of 100-B. > > You can also get some marginal return by reducing database in-memory > limits, which control the sizes of in-memory stands. But there isn't more > than about half a GB on the table there, and ingestion performance may > suffer. If you make the limits too small, some documents may not ingest. > > You could try to minimize your group-level caches. That's a somewhat > dangerous game, since trimming them too much will result in > EXPNTREECACHEFULL and LISTCACHEFULL errors. Those caches generally use > about 3/8 of RAM, so this can be a *temporary* way to get a little > breathing room while you make other adjustments. But the system is designed > to use those caches, and won't perform well if they are too small. > > -- Mike > > On 4 Nov 2011, at 08:10 , John Zhong wrote: > > > Thank you, Mike. And yes, I knew the server was overloaded. I just want > to know if there is another suggestion except upgrading the server. It is > probably no. > > > > Thanks again. > > John > > > > On Fri, Nov 4, 2011 at 11:00 PM, Michael Blakeley <[email protected]> > wrote: > > To be blunt: that host is out of memory and over capacity. > > > > Stop ingestion right away, and revert the configuration changes you have > made. Turning off merges is almost always a mistake: there ought to be a > pop quiz built into the configuration screen. You can easily paint yourself > into a corner. The other changes were almost certainly counterproductive, > too. > > > > How can I tell that you are out of memory? The error message states that > MarkLogic was unable to map a 5-GiB index. That is only one of many indexes > that need to memory-map, so I would guess that the system is badly > overloaded. The rest of the data confirms it: on a 16-GiB host your total > index memory (aka forest memory) is about 12-GiB. You can't use all the > host memory for forest indexes: you'll need about that much again for > group-level caches, and about 25% of RAM for queries, merges, and the OS. > For your data so far, you ought to have at least 32-GB RAM. > > > > Do you know where you are in your total load? Are you about 10% done, or > closer to 75% done? > > > > With the system in its current state, it will probably use a lot of > virtual memory for paging, and performance will probably be unacceptable. > Am I right in thinking this is a VM? If so, you may be able to upgrade the > VM memory without much trouble. Virtual or not, if a memory upgrade isn't > practical then I would recommend clearing the database and starting over, > and this time stop before you are out of memory. > > > > -- Mike > > > > On 4 Nov 2011, at 00:33 , John Zhong wrote: > > > > > Hi all, > > > > > > I am running MarkLogic 4.1-6 version and encountered the below error > today: > > > > > > XDMP-FORESTERR: Error in merge of forest: SVC-MAPBIG: Mapped file too > large to map: 5825151760 bytes: 'D:\Program > Files\MarkLogic\Data\Forests\forest1\00001cde\580f5ba44789ea78_dateTime' > > > > > > The MarkLogic instance is running on a EC2 server has 1 CUP with 2 > cores, 17.1GB RAM, 64 bit windows server 2008. We have created only one > database with one forest, now the forest has 6 stands, total is 86GB, and > there is still 300GB disk space available. > > > > > > active fragments deleted fragments on disk > size in memory size > > > stand 1: 2,922,206 3,320,621 36,246 > MB 4,162 MB > > > stand 2: 1,111,868 1,795,830 39,987 > MB 6,836 MB > > > stand 3: 63,545 107,616 > 3,588 MB 606 MB > > > stand 4: 7,620 24,492 > 925 MB 163 MB > > > stand 5: 5,638 6,882 > 479 MB 79 MB > > > stand 6: 11,757 28,626 > 1,761 MB 319 MB > > > TOTAL: 4,122,634 5,284,067 82,986 > MB 12,165 MB > > > > > > We enable the default merge setting and tried to modify the merge max > size to 2048 then merge manually, but still failed. > > > > > > Now, we disable the merge. > > > > > > Any sugestion will be very appreciate!! > > > > > > Thank you in advance. > > > John > > > > > > > > > _______________________________________________ > > > General mailing list > > > [email protected] > > > http://developer.marklogic.com/mailman/listinfo/general > > > > _______________________________________________ > > General mailing list > > [email protected] > > http://developer.marklogic.com/mailman/listinfo/general > > > > _______________________________________________ > > General mailing list > > [email protected] > > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general >
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
