Well we have a 4-node farm and one of these is currently taken out of the farm. This one is currently running in "broken" mode.
I will see if Process Explorer throws anything up. Thanks On 23 April 2012 14:49, Simon Svensson <[email protected]> wrote: > I believe you've configured your debugger to break at first exception, so > the dump won't show anything else. > > I would wait until the problem appears again and look into different > performance counters. I've seen similar scenarios with memory leaks where > "% time in gc" (category .net clr memory) indicates that it spends a lot of > time collecting garbage. This would, depending on the gc implementation in > use (I don't recall which one IIS uses by default) either slow down your > application, or completely pause it while collecting occurs, which would > match what you describe. > > You should also look into cpu usage and memory usage. The kind of memory > leak I'm talking about would cause high amount of cpu usage (garbage > collection) and high amount of memory usage, usually enough to start > swapping away like crazy. > > You could use a tool like Process Explorer (from Sysinternals) to generate > a dump from a running program, and windbg to dive into the generated dump. > > There's nothing so far that indicates that Lucene is the culprit. Are you > using any other non-bcl libraries? > > On 23 apr 2012, at 15:37, Alastair Todd wrote: > > > OK< sorry for the double post - wasnt sure if the first one went through > as > > it was before I'd subscribed. > > > > Anyway, I don't think its a prob posting the dmp - if i do in fact have > > this. Is this the .dmp file generated by the debug diagnostic tool? If so > > its like 186Mb zipped. > > > > The site stops responding to requests eventually (browser sits > > there indefinitely loading) and only an app pool recycle fixes it. And > > we've no set of actions to manually trigger the "crash". > > > > > > > > On 23 April 2012 14:26, Simon Svensson <[email protected]> wrote: > > > >> Hi, > >> > >> This is a common exception which is handled within the QueryParser > itself. > >> You're seeing a first chance exception which does not cause the crashes > >> you're experiencing. > >> > >> Is this from a production system? Can you share the raw memory dump, or > >> does it contain sensitive information? > >> > >> // Simon > >> > >> On 23 apr 2012, at 15:21, Alastair Todd wrote: > >> > >>> I have been investigating an IIS 7.5 crashing website which only gets > >> fixed > >>> by an app pool recycle. > >>> > >>> I have used DebugDiag to catch crash exceptions. The problem appears to > >> be > >>> manifesting itself in Lucene.Net. > >>> > >>> I am confused as to why this would fatally crash IIS. Should we be > >> wrapping > >>> a try catch around the call and throwing our own exception? > >>> > >>> Any "normal" website exception would normally be bubbled up and logged > by > >>> ELMAH, but this error never finds its way into the ELMAH log (makes > sense > >>> since it crashes IIS). > >>> > >>> How would you go about keeping this site up? > >>> > >>> Here's the report: > >>> > >>> Report for W3WP__~1.DMP > >>> Type of Analysis Performed Crash Analysis > >>> Machine Name WEB01 > >>> Operating System Windows Server 2008 Service Pack 2 > >>> Number Of Processors 8 > >>> Process ID 20048 > >>> Process Image C:\Windows\SysWOW64\inetsrv\w3wp.exe > >>> System Up-Time 214 day(s) 23:40:43 > >>> Process Up-Time 00:44:32 > >>> > >>> > >>> Thread 34 - System ID 22368 > >>> Entry point mscorwks!Thread::intermediateThreadProc > >>> Create time 23/04/2012 11:49:25 > >>> Time spent in user mode 0 Days 0:0:16.582 > >>> Time spent in kernel mode 0 Days 0:0:1.622 > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> .NET Call Stack > >>> > >>> > >>> > >>> Function > >>> Lucene.Net.QueryParsers.FastCharStream.Refill() > >>> Lucene.Net.QueryParsers.FastCharStream.ReadChar() > >>> Lucene.Net.QueryParsers.QueryParserTokenManager.JjMoveNfa_3(Int32, > Int32) > >>> > >> > Lucene.Net.QueryParsers.QueryParserTokenManager.JjMoveStringLiteralDfa0_3() > >>> Lucene.Net.QueryParsers.QueryParserTokenManager.GetNextToken() > >>> Lucene.Net.QueryParsers.QueryParser.Jj_scan_token(Int32) > >>> Lucene.Net.QueryParsers.QueryParser.Jj_3R_2() > >>> Lucene.Net.QueryParsers.QueryParser.Jj_3_1() > >>> Lucene.Net.QueryParsers.QueryParser.Jj_2_1(Int32) > >>> Lucene.Net.QueryParsers.QueryParser.Clause(System.String) > >>> Lucene.Net.QueryParsers.QueryParser.Query(System.String) > >>> Lucene.Net.QueryParsers.QueryParser.TopLevelQuery(System.String) > >>> Lucene.Net.QueryParsers.QueryParser.Parse(System.String) > >>> > >> > OurProject.Content.Search.SearchManager.GetModuleSearchResults(System.String) > >>> Search.Page_Load(System.Object, System.EventArgs) > >>> System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, > >>> System.Object, System.Object, System.EventArgs) > >>> System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, > >>> System.EventArgs) > >>> System.Web.UI.Control.OnLoad(System.EventArgs) > >>> System.Web.UI.Control.LoadRecursive() > >>> System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean) > >>> System.Web.UI.Page.ProcessRequest(Boolean, Boolean) > >>> System.Web.UI.Page.ProcessRequest() > >>> System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext) > >>> System.Web.UI.Page.ProcessRequest(System.Web.HttpContext) > >>> ASP.search_aspx.ProcessRequest(System.Web.HttpContext) > >>> > >> > System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() > >>> System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) > >>> > >> > System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception) > >>> > >> > System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, > >>> System.Web.HttpContext) > >>> > >> > System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, > >>> IntPtr, IntPtr, Int32) > >>> System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, > >>> System.Web.RequestNotificationStatus ByRef) > >>> > >> > System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, > >>> IntPtr, IntPtr, Int32) > >>> System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, > >>> IntPtr, IntPtr, Int32) > >>> > >>> > >>> Full Call Stack > >>> > >>> > >>> > >>> Function Arg 1 Arg 2 Arg 3 Arg 4 Source > >>> kernel32!RaiseException+58 e0434f4d 00000001 00000001 > >>> 1cc0ec08 > >>> mscorwks!RaiseTheExceptionInternalOnly+2a8 29af23c4 00000000 > >>> 00000000 c22aa5e5 > >>> mscorwks!JIT_Throw+fc 0000000f 29aefc04 29af12f0 > 7fffffff > >>> Lucene.Net.QueryParsers.FastCharStream.Refill() c22aa4c1 > >>> 00000000 29af12f0 00000000 > >>> mscorwks!JIT_NewArr1+1e3 00000000 29aef878 1cc0ee20 > >> 1cc0ede8 > >>> Lucene.Net.QueryParsers.QueryParser.Jj_scan_token(Int32) 29aef878 > >>> 1cc0edf8 017167c5 29af1310 > >>> Lucene.Net.QueryParsers.QueryParser.Jj_3R_2() 29af1310 > >>> 29aef878 1cc0ee30 0171672c > >>> Lucene.Net.QueryParsers.QueryParser.Jj_3_1() 29af1310 29aef878 > >>> 00000000 00000002 > >>> Lucene.Net.QueryParsers.QueryParser.Jj_2_1(Int32) 29aef878 > >>> 29af1310 00000000 00000000 > >>> Lucene.Net.QueryParsers.QueryParser.Clause(System.String) 29aef1e8 > >>> 29ab4cf0 29aeb9a8 1cc0ef70 > >>> System.Web.UI.StateBag.get_Item(System.String) 0c9955e8 > >>> 1cc0ef94 6ff33d94 0c9955e8 > >>> System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, > >>> System.Object, System.Object, System.EventArgs) 0c9955e8 > >>> 29ab4cf0 29ab4cf0 29adf078 > >>> System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, > >>> System.EventArgs) 0c9955e8 29ab4cf0 29ab4cf0 1cc0f108 > >>> System.Web.UI.Control.OnLoad(System.EventArgs) 01eda6a4 > >>> 29ab4cf0 29ab4cf0 1cc0f108 > >>> System.Web.UI.Control.LoadRecursive() 00000000 0a9901d0 > >>> 00000000 29a6600c > >>> System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean) 00000001 > >>> 29ab4cf0 00000001 00000000 > >>> System.Web.UI.Page.ProcessRequest(Boolean, Boolean) 00000001 > >>> 0e9e76bc 0e9e76bc 0c990f4c > >>> System.Web.UI.Page.ProcessRequest() 29ab4cf0 1cc0f1ac > >>> 6ff28e32 00000000 > >>> System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext) > >>> 00000000 29ab4cf0 29a6ac0c 1cc0f1e4 > >>> System.Web.UI.Page.ProcessRequest(System.Web.HttpContext) 0250af40 > >>> 00000000 29a6600c 00000000 > >>> > >> > System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() > >>> 00000000 00000000 00000000 29a6ac0c > >>> System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) > >>> 1cc0f280 8c16dcd2 72e149e8 1cc0f2d8 > >>> > >> > System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception) > >>> 00000000 00000000 1cc0f2d8 1cc0f334 > >>> > >> > System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, > >>> System.AsyncCallback) 0e99b5e8 8c16dcd2 72e149e8 > >>> 1cc0f344 > >>> > >> > System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, > >>> System.Web.HttpContext) 29a6600c 8c16dcd2 72e149e8 > >>> 1cc0f898 > >>> > >> > System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, > >>> IntPtr, IntPtr, Int32) 00000000 00000009 00000000 > >>> 00000000 > >>> System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, > >>> IntPtr, IntPtr, Int32) 01a6119c 023d15a0 00000009 > >>> 00000000 > >>> webengine!W3_MGD_HANDLER::ProcessNotification+59 023d0ab8 > >>> 023d15a0 00000000 1cc0f46c > >>> webengine!W3_MGD_HANDLER::DoWork+235 00000080 00000000 > >>> 1b7233bc 023d0b10 > >>> webengine!RequestDoWork+2c3 00000080 00000000 1b7233bc > >>> 023d0ab8 > >>> webengine!CMgdEngHttpModule::OnExecuteRequestHandler+18 023d0ab8 > >>> 023d0b10 023d0abc 1cc0f518 > >>> iiscore!NOTIFICATION_CONTEXT::RequestDoWork+128 1b7233bc > >>> 023d0ab8 00000000 023d0b10 > >>> iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+2ec 00000000 > >>> 00000000 00000000 023d0ab8 > >>> iiscore!NOTIFICATION_CONTEXT::CallModules+28 00000000 00000000 > >>> 00000000 023d0ab8 > >>> iiscore!W3_CONTEXT::DoStateRequestExecuteHandler+36 00000000 > >>> 00000000 00000000 00000000 > >>> iiscore!W3_CONTEXT::DoWork+d7 00000000 00000000 00000001 > >>> 00000000 > >>> iiscore!W3_MAIN_CONTEXT::ContinueNotificationLoop+1f 00000000 > >>> 00000000 023d0ab8 1cc0f83c > >>> iiscore!W3_MAIN_CONTEXT::ProcessIndicateCompletion+1f 00000000 > >>> 023d15a0 767516b3 023d0ab8 > >>> iiscore!W3_CONTEXT::IndicateCompletion+63 00000000 1cc0f8e8 > >>> 1cc0f8ac 1cc0f860 > >>> webengine!W3_MGD_HANDLER::IndicateCompletion+3b 00000000 > >>> 1cc0f898 1cc0f880 01771671 > >>> webengine!MgdIndicateCompletion+22 023d15a0 1cc0f8e8 > >>> 8c16dcd2 00000000 > >>> 0x01771671 29a6600c 29a65dc4 00000000 1cc0f91c > >>> > >> > System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, > >>> IntPtr, IntPtr, Int32) 00000004 00000003 00000000 > >>> 00000000 > >>> System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, > >>> IntPtr, IntPtr, Int32) 00000000 023d15a0 00000003 > >>> 00000004 > >>> mscorwks!UM2MThunk_WrapperHelper+1f 1cc0fb30 00000010 > >>> 01682388 016bee48 > >>> mscorwks!UM2MThunk_WrapperWorker+60 1cc0facc 0250af40 > >>> 0250af40 1cc0faa8 > >>> mscorwks!UM2MThunk_Wrapper+14 1cc0facc 1cc0fa9c 72ef7e43 > >>> c22ab3b9 > >>> mscorwks!Thread::DoADCallBack+da 00000003 7308e32d > >>> 1cc0facc c22ab211 > >>> mscorwks!UM2MDoADCallBack+7c ffffffff 0250af40 1cc0fb30 > >>> 00000010 > >>> 0x01682372 00000000 023d15a0 00000003 00000004 > >>> webengine!W3_MGD_HANDLER::ProcessNotification+59 7336d32c > >>> 1cf27800 1cc0fba8 72e54404 > >>> webengine!ProcessNotificationCallback+2d 023d15a0 c22ab2b9 > >>> 72e53d1c 00040000 > >>> mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+9a > >>> 00000000 00000000 76751110 00000001 > >>> mscorwks!ThreadpoolMgr::ExecuteWorkRequest+af c22ab505 > >>> 00000000 72e53d69 0250c220 > >>> mscorwks!ThreadpoolMgr::WorkerThreadStart+20b 00000000 > >>> fffde000 77c70094 1cc0fbb8 > >>> mscorwks!Thread::intermediateThreadProc+49 0250c220 1cc0ff7c > >>> 77c0d80d 0250c220 > >>> kernel32!BaseThreadInitThunk+e 0250c220 39fb7237 00000000 > >>> 00000000 > >>> ntdll!__RtlUserThreadStart+23 72f76e83 0250c220 ffffffff > >>> 77c2e348 > >>> ntdll!_RtlUserThreadStart+1b 72f76e83 0250c220 00000000 > >>> 00000000 > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> Exception Information > >>> In W3WP__~1.DMP the assembly instruction at kernel32!RaiseException+58 > >>> in C:\Windows\SysWOW64\kernel32.dll from Microsoft Corporation has > >>> caused a CLR Exception on thread 34 with the following error > >>> information: > >>> > >>> Type: System.IO.IOException > >>> Message: read past eof > >>> > >>> > >>> This exception originated from > >> Lucene.Net.QueryParsers.FastCharStream.Refill(). > >> > >> > >
