This is 64 bit RHEL 4.7... We did have lots of MEMALLOC issues with 32-bit and extended memory, but moving to 64-bit resolved those...
On 6/30/2010 11:35 AM, Lee, David wrote: > Are you running 32 or 64 bit OS ? > My (minimal) experience is that 32 bit OS with MLS can be unstable at > times causing these same problems. > Feedback from ML is that memory can get fragmented and that this issue > goes away on 64 bit OS. > Never had the problem on my Amazon EC2 instances (which are 64 bit OS) > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Wyatt > VanderStucken > Sent: Wednesday, June 30, 2010 8:30 AM > To: [email protected] > Subject: Re: [MarkLogic Dev General] Linux Huge Pages? > > Hi Danny, > > Thanks for responding. I had found that info about the Huge Pages on > the xqzone site after I wrote originally. I've since put it into place > on this particular machine - and it might have improved things. > Maybe... > > This server has always had freezing issues that seem to be related to > the error in MLS (the only thing running on the box). It's always been > intermittent (sometimes months apart, sometimes repetitively for several > > consecutive days) and I've never been sure if it's the freezing that > causes issues in MLS or issues in MLS that cause the freezing. The > entire machine (except RAM) has been swapped and a clean RedHat > installation has occurred... > > It's only a dev server and our production servers hum along just fine. > I have to live 4.1-5 until September (at least). Hopefully the Huge > Pages will improve things or it will be months again before the next > episode. Let me know if you have any ideas about how to troubleshoot > it. > > Regards, > Wyatt > > > On 6/29/2010 6:31 PM, Danny Sokolsky wrote: >> Hi Wyatt, >> >> I am not sure, but I think that crash was fixed in 4.1-6. I am not > sure if it is related to your huge page setting being less than > recommended or not. You might want to try this on 4.1-6 and see if it > is fixed. >> Here is what the MarkLogic Installation Guide says about huge pages > (including a link to a Red Hat article about setting it): >> * On Linux systems, Mark Logic recommends setting Linux Huge > Pages to 3/8 the size of your physical memory. For details on setting up > Huge Pages, see > http://magazine.redhat.com/2007/05/29/how-do-i-set-up-hugepages-in-red-h > at-enterprise-linux-4/. >> If you have Huge Pages set up on a Linux system, your swap space > on that machine should be twice the size of your physical memory minus > the size of your Huge Page (because Linux Huge Pages are not swapped). > For example, if you have 64GB of physical memory, and if you have Huge > Pages set to 24 GB, then you need swap space of 80 GB ([64 - 24] x 2). >> At system startup on Linux machines, MarkLogic Server logs a > message to the ErrorLog.txt file showing the Huge Page size, and the > message indicates if the size is below the recommended level. >> >> -Danny >> >> -----Original Message----- >> From: [email protected] > [mailto:[email protected]] On Behalf Of Wyatt > VanderStucken >> Sent: Tuesday, June 29, 2010 7:46 AM >> To: General Mark Logic Developer Discussion >> Subject: [MarkLogic Dev General] Linux Huge Pages? >> >> Greetings, >> >> I have a server which keeps crashing, and I'm wondering if the log >> message "Linux Huge Pages: detected 0, recommend 1280 to 1971" has >> anything to do with it. I'm not familiar w/ the term "" - can anyone >> give me some idea if this is at all related. I've included the full >> stack trace below. >> >> Regards, >> Wyatt >> >> >> >> >> >> >> 2010-06-29 10:29:11.477 Info: Linux Huge Pages: detected 0, recommend >> 1280 to 1971 >> Segmentation fault in thread 1101044064 >> Thread 14 (Thread 1075833184 (LWP 14622)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 13 (Thread 1078053216 (LWP 14624)): >> #0 0x0000003e5a70b55f in __accept_nocancel () from >> /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd72cb in svc::Socket::accept () >> #2 0x0000000000b39407 in xdmp::AppServerThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 12 (Thread 1080146272 (LWP 14625)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 11 (Thread 1080273248 (LWP 14626)): >> #0 0x0000003e5a70b55f in __accept_nocancel () from >> /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd72cb in svc::Socket::accept () >> #2 0x0000000000b39407 in xdmp::AppServerThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 10 (Thread 1082366304 (LWP 14627)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 9 (Thread 1082493280 (LWP 14628)): >> #0 0x0000003e5a70b55f in __accept_nocancel () from >> /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd72cb in svc::Socket::accept () >> #2 0x0000000000b39407 in xdmp::AppServerThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 8 (Thread 1083267424 (LWP 14631)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 7 (Thread 1107581280 (LWP 14647)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 6 (Thread 1094506848 (LWP 14652)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 5 (Thread 1101044064 (LWP 14659)): >> #0 0x0000003e5a70c18a in waitpid () from /lib64/tls/libpthread.so.0 >> #1 0x000000000168d602 in xdmp::pstack () >> #2 0x000000000168d6e8 in sigsegvHandler () >> #3<signal handler called> >> #4 0x0000000001c603ce in xdmp::ForClause::flworCollectOrderingClauses > () >> #5 0x0000000001c3c759 in xdmp::AxisStepExpr::~AxisStepExpr$delete () >> #6 0x0000000001c401b3 in xdmp::PathExpr::~PathExpr$delete () >> #7 0x0000000001c3b793 in >> xdmp::ElementContentList::~ElementContentList$delete >> #8 0x0000000001c11d05 in >> xdmp::ElementConstructor::~ElementConstructor$delete >> #9 0x0000000001bf1785 in xdmp::IfExpr::~IfExpr$delete () >> #10 0x0000000001bf1785 in xdmp::IfExpr::~IfExpr$delete () >> #11 0x0000000001c3b793 in >> xdmp::ElementContentList::~ElementContentList$delete >> #12 0x0000000001c11d05 in >> xdmp::ElementConstructor::~ElementConstructor$delete >> #13 0x0000000001c3b793 in >> xdmp::ElementContentList::~ElementContentList$delete >> #14 0x0000000001c11d05 in >> xdmp::ElementConstructor::~ElementConstructor$delete >> #15 0x0000000001c3b793 in >> xdmp::ElementContentList::~ElementContentList$delete >> #16 0x0000000001c11d05 in >> xdmp::ElementConstructor::~ElementConstructor$delete >> #17 0x0000000001c3eae9 in xdmp::FLWORExpr::~FLWORExpr$delete () >> #18 0x0000000001beb483 in xdmp::FunDecl::~FunDecl$delete () >> #19 0x0000000001c3db47 in xdmp::Prolog::~Prolog$delete () >> #20 0x0000000001c68d2c in xdmp::Module::~Module$base () >> #21 0x0000000001c690eb in xdmp::LibraryModule::~LibraryModule$delete > () >> #22 0x0000000001c8b108 in xdmp::Statement::~Statement$delete () >> #23 0x0000000001c8f5c8 in xdmp::Program::~Program$delete () >> #24 0x0000000000b5c0fa in svc::Handle<xdmp::Program>::operator= () >> #25 0x0000000001cad72b in svc::Hash64Cache<svc::Handle<xdmp::Program> >> >::clear >> #26 0x0000000001c8684d in xdmp::ProgramCache::_clear () >> #27 0x0000000001c868a5 in xdmp::ProgramCache::clear () >> #28 0x0000000000b17962 in xdmp::AppServer::updateConfig () >> #29 0x00000000016de0b0 in xdmp::Server::updateConfigServers () >> #30 0x00000000016e14ff in xdmp::Server::updateConfigServer () >> #31 0x00000000016e3f4e in xdmp::ServerConfig::loadConfig () >> #32 0x0000000000c2df39 in xdmp::Config::writeFile () >> #33 0x0000000000ae724d in xdmp::xdmp_write_cluster_config_file () >> #34 0x0000000000bdb894 in xdmp::TwoArgBuiltinExpr::_eval () >> #35 0x0000000000be7613 in xdmp::BuiltinEnv::eval () >> #36 0x0000000000be903d in xdmp::Value::builtinArg () >> #37 0x0000000000be9146 in xdmp::Value::builtinArg () >> #38 0x0000000000beebc2 in xdmp::BuiltinExpr::eval () >> #39 0x0000000001ce7843 in xdmp::LetClause::flworEval () >> #40 0x0000000001cd24c5 in xdmp::FLWORExpr::eval () >> #41 0x0000000001cb46e5 in xdmp::Expr::evalCompletely () >> #42 0x0000000000e55358 in xdmp::TryCatchExpr::eval () >> #43 0x0000000001ce2091 in xdmp::FunctionEnv::eval () >> #44 0x0000000001ce4945 in xdmp::Value::functinArg () >> #45 0x0000000001ce4be8 in xdmp::ValueSequence::functinArg () >> #46 0x0000000001ce45e7 in xdmp::Value::functinArg () >> #47 0x0000000001ce45e7 in xdmp::Value::functinArg () >> #48 0x0000000001ce45e7 in xdmp::Value::functinArg () >> #49 0x0000000001ce34f4 in xdmp::FunctionCall::eval () >> #50 0x0000000001ce4e45 in xdmp::ExprSequence::eval () >> #51 0x0000000001cb5f76 in xdmp::IfExpr::eval () >> #52 0x0000000001cb5f76 in xdmp::IfExpr::eval () >> #53 0x0000000001cb5f76 in xdmp::IfExpr::eval () >> #54 0x0000000001cb4db3 in xdmp::FLWORExpr::_flworEval () >> #55 0x0000000001ce7ec5 in xdmp::LetClause::flworEval () >> #56 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #57 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #58 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #59 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #60 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #61 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #62 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #63 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #64 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #65 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #66 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #67 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #68 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #69 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #70 0x0000000001cd24c5 in xdmp::FLWORExpr::eval () >> #71 0x0000000001cb5f76 in xdmp::IfExpr::eval () >> #72 0x0000000001cb4db3 in xdmp::FLWORExpr::_flworEval () >> #73 0x0000000001ce7ec5 in xdmp::LetClause::flworEval () >> #74 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #75 0x0000000001ce78e7 in xdmp::LetClause::flworEval () >> #76 0x0000000001cd24c5 in xdmp::FLWORExpr::eval () >> #77 0x0000000001ccac3a in xdmp::StatementExpr::eval () >> #78 0x0000000001c822f9 in xdmp::Statement::evalQuery () >> #79 0x0000000001c9898f in xdmp::Statement::eval () >> #80 0x0000000000b48ff0 in xdmp::AppConnectionTask::handleEvalLocked () >> #81 0x0000000000b4c103 in xdmp::AppConnectionTask::handleEval () >> #82 0x0000000000b4e7a8 in xdmp::AppConnectionTask::handleFile () >> #83 0x0000000000b534f9 in xdmp::AppConnectionTask::handleConnection () >> #84 0x0000000000b552cd in xdmp::AppConnectionTask::run () >> #85 0x0000000001de8720 in svc::PooledThread::run () >> #86 0x0000000001de52b9 in svc::Thread::top () >> #87 0x0000000001de6919 in runThread () >> #88 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #89 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 4 (Thread 1107839328 (LWP 14661)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 3 (Thread 1077926240 (LWP 14662)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 2 (Thread 1098951008 (LWP 14663)): >> #0 0x0000003e5a70a7ef in sem_wait () from /lib64/tls/libpthread.so.0 >> #1 0x0000000001dd1b3a in svc::Semaphore::wait () >> #2 0x0000000001de84d7 in svc::PooledThread::run () >> #3 0x0000000001de52b9 in svc::Thread::top () >> #4 0x0000000001de6919 in runThread () >> #5 0x0000003e5a706137 in start_thread () from > /lib64/tls/libpthread.so.0 >> #6 0x0000003e59cc9883 in clone () from /lib64/tls/libc.so.6 >> Thread 1 (Thread 182921705920 (LWP 14615)): >> #0 0x0000003e5a70bc35 in __nanosleep_nocancel () >> #1 0x0000000001de470d in svc::Thread::sleep () >> #2 0x00000000016a77c3 in xdmp::Server::background () >> #3 0x00000000016ae4de in xdmp::run () >> #4 0x00000000016b28f4 in xdmp::Server::main () >> #5 0x0000003e59c1c40b in __libc_start_main () from > /lib64/tls/libc.so.6 >> #6 0x000000000040ddea in _start () >> #7 0x0000007fbffff9d8 in ?? () >> #8 0x000000000000001c in ?? () >> #9 0x0000000000000001 in ?? () >> #10 0x0000007fbffffbd1 in ?? () >> #11 0x0000000000000000 in ?? () >> 2010-06-29 10:34:00.076 Notice: Starting MarkLogic Server 4.1-5 x86_64 >> in /opt/MarkLogic with data in /apps/MarkLogic/Data >> 2010-06-29 10:34:00.076 Info: Copyright 2002-2010 Mark Logic >> Corporation. All Rights Reserved. >> 2010-06-29 10:34:00.076 Info: Contains LinguistX from Inxight > Software, >> Inc. Copyright 1996-2006. All rights reserved. www.inxight.com >> 2010-06-29 10:34:00.076 Info: This product includes software developed >> by the OpenSSL Project for use in the OpenSSL Toolkit >> (http://www.openssl.org/) >> 2010-06-29 10:34:00.076 Info: Copyright (C) 1995-1998 Eric Young >> ([email protected]) All rights reserved. >> 2010-06-29 10:34:00.076 Info: Copyright (C) 1998-2001 The OpenSSL >> Project. All rights reserved. >> 2010-06-29 10:34:00.085 Info: Host eng018.ets.org running Linux >> 2.6.9-78.ELsmp on Xen (Red Hat Enterprise Linux AS release 4 (Nahant >> Update 7)) >> 2010-06-29 10:34:00.706 Info: Mounted forest Schemas locally on >> /apps/MarkLogic/Data/Forests/Schemas >> 2010-06-29 10:34:00.722 Info: Mounted forest App-Builder locally on >> /apps/MarkLogic/Data/Forests/App-Builder >> 2010-06-29 10:34:00.727 Info: Mounted forest Modules locally on >> /apps/MarkLogic/Data/Forests/Modules >> 2010-06-29 10:34:00.744 Info: Mounted forest Documents locally on >> /apps/MarkLogic/Data/Forests/Documents >> 2010-06-29 10:34:00.751 Info: Mounted forest Security locally on >> /apps/MarkLogic/Data/Forests/Security >> 2010-06-29 10:34:00.759 Info: Mounted forest Last-Login locally on >> /apps/MarkLogic/Data/Forests/Last-Login >> 2010-06-29 10:34:00.765 Info: Mounted forest Triggers locally on >> /apps/MarkLogic/Data/Forests/Triggers >> 2010-06-29 10:34:00.826 Info: Linux Huge Pages: detected 0, recommend >> 1280 to 1971 >> >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
