Hi Tamas, Since you have access to a Windows Server 2003 x64 R2 machine, can you tell me your experience running MapServer on it? Did you experience random memory corruption the way I did?
Hong On Thu, Feb 15, 2007 at 11:17:45AM +0100, Tamas Szekeres wrote: > Hi Hong, > > Are you sure you've made an x64 build for your dependent libraries? > You should start by enabling only the vital components in the build, > and adding the new dependencies one by one after a successful build. > The gd library is a vital component for mapserver. I would suggest to > set up a configuration enabling only the gd component in nmake.opt. > > I've attached my mapserver nmake.opt and the gd makefile.vc. I've > tested these configurations on a Windows Server 2003 x64 R2 and AMD64. > > You might also compile the SDE related stuff or obtain an x64 > compatible library version for linking against. > > > Best regards, > > Tamas > > > > 2007/2/15, Hong <[EMAIL PROTECTED]>: > >Ok, trying to compile everything in pure 64-bit mode, compilation went > >fine up > >to the moment of linking: > > > >link /dll mapbits.obj maphash.obj mapshape.obj mapxbase.obj... > > > >that ended up with symbol referencing error: > > > >... > >Creating library mapserver_i.lib and object mapserver_i.exp > >mapraster.obj : error LNK2001: unresolved external symbol gdImageDestroy > >mapgd.obj : error LNK2001: unresolved external symbol gdImageDestroy > >... > >mapsde.obj : error LNK2019: unresolved external symbol SE_reginfo_free > >referenced > >in function msSDELayerGetRowIDColumn > >mapsde.obj : error LNK2019: unresolved external symbol > >SE_reginfo_get_rowid_column referenced in function msSDELayerGetRowIDColumn > >... > > > >All the libs are in correct places, location hard coded, and passed to > >LINK. > >Searching on google revealed that someone had similar problem compiling > >GDAL > >on 64-bit as well. The message on gdal-dev mentioned about the way symbol > >names are decorated differently in 64-bit windows. I'm thinking bgd.def > >might > >have trailing "@nn" that needs to be replaced with something else. Does > >anyone > >have any experience dealing with this issue? > > > >The original message on gdal-dev: > > > >http://lists.maptools.org/pipermail/gdal-dev/2006-May/008885.html > > > >Thanks. > > > >Hong > > > > > >On Fri, Feb 02, 2007 at 10:51:21PM +0100, Tamas Szekeres wrote: > >> Since this issue seems to be quite an essential it would be useful to > >> keep it tracked through a bug report. Would you submit one by > >> summarizing the all the information we currently have > >> (http://mapserver.gis.umn.edu/bugs/enter_bug.cgi)? Assign the bug to > >> the C# binding. > >> > >> Here are some further tests that would worth trying: > >> > >> 1. Try to alter the SDE connection pooling behaviour. Adding > >> PROCESSING "CLOSE_CONNECTION=DEFER" would set it on a per layer > >> basis. I don't know exactly which would be the best at this case, but > >> in fact the connections are retained per threads. Maybe keeping the > >> connections alive would affect how often the memory allocations > >> related to the connections take place. > >> > >> 2. I've just added a possibility to use a global lock when invoking > >> into the mapscript code. To enable this option you should uncomment > >> #define USE_GLOBAL_LOCK in csmodule.i. and recompile the mapscript > >> code. This setting would ensure that only one thread will invoke into > >> libmap.dll in most cases simultaneously, and could help detecting some > >> threading issues, though this solution is not equal as restricting the > >> invocation to one thread (like the COM STA model would do for > >> example). > >> > >> 3. Ensure disposing the cloned mapObj explicitly by the "using" > >> directive for example, like > >> > >> using (map = ((mapObj)Session("m_mapObj")).clone() ) > >> { > >> // operate on map > >> } > >> > >> The child references (if any) should also be controlled by an internal > >> using blocks. All of the mapscript objects implement IDisposable. > >> > >> > >> Best regards, > >> > >> Tamas > >> > >> > >> > >> 2007/2/2, Hong <[EMAIL PROTECTED]>: > >> >On Thu, Feb 01, 2007 at 10:47:00PM +0100, Tamas Szekeres wrote: > >> >> Hi Hong, > >> >> > >> >> Are you still getting the problem at the same function or the location > >> >> may change in a random fashion? Do you have a stacktrace on it? > >> > > >> >I'll summarize the various states of MapServer for me so far: > >> > > >> >VS2003 compilation on 32-bit environment: stable (no memory corruption > >at > >> >all) > >> > > >> >VS2005 compilation on 32-bit environment: behavior is unpredictable, > >memory > >> >corruption at will, be that assigning filter string to layer, creating a > >> >new > >> >classification, etc. > >> > > >> >VS2003 compilation on 64-bit environment: 100% memory corruption. > >> > > >> >VS2005 compilation on 64-bit environment: behavior is unpredictable, > >memory > >> >corruption at will. As a matter of curiosity, this happened: > >> >With debugger attached to the main process running MapServer, I went to > >the > >> >'immediate window' after a breakpoint and did > >> >'?p_classObj.getExpressionString' a few times. The first few times, I > >got > >> >the > >> >expression string, on the 4th time or so, access violation happened. > >> >Hmmm... > >> > > >> >A chunk of my project requires MapServer to run reliably on 64-bit > >> >environment, seems like that is a no-go for any foreseeable future. > >Worst > >> >of > >> >all, when memory corruption happened with MapServer, the memory space of > >> >ASP.NET worker process is messed up as well. Only through restarting the > >> >worker process is the process back to its "normal" self. > >> > > >> >However, I haven't tried hard compiling MapServer in truly 64-bit way, > >that > >> >is, through the VS2005 64-bit command prompt. Tried a few times before, > >but > >> >failed miserably. This is what I will do next in my attempt to get it to > >> >run > >> >reliably on 64-bit environment. > >> > > >> >> At the moment I can think of 3 issues the memory problem might be > >related > >> >> to. > >> >> > >> >> 1. Usage of different CRT heaps / using different dll versions > >> >> You should make sure (once again) that only 1 version of the CRT dll > >> >> is used in the dependency tree of mapscript.dll. In this case > >> >> msvcr71.dll would be the expected version. If the same dll is > >> >> referenced from multiple dll-s make sure that the references point to > >> >> the same location. Make sure that the same dll has been loaded as you > >> >> have compiled against. > >> >> Kicking off the unused dependencies are also a good practice. (May be > >> >> you have possibly done it) > >> > > >> >Double-checked all the dependencies. All the DLLs use the same CRT > >library. > >> > > >> >> 2. Early garbage collection > >> >> Ensure that the lifespan of the child references (eg. layerObj) are > >> >> not falling outside of the lifespan of the root object (eg. mapObj). > >> >> Here is an example may cause potential problems: > >> >> > >> >> parentObj parent = new parentObj(); > >> >> childObj child = parent.getChildObj(); > >> >> // at this point there's no more references to 'parent' so it might be > >> >> garbage collected > >> >> child.OperateOnChild(); // will cause access violation > >> >> > >> >> So as to keep the parent alive until a certain point you might use > >> >> GC.KeepAlive() as. > >> >> > >> >> parentObj parent = new parentObj(); > >> >> childObj child = parent.getChildObj(); > >> >> child.OperateOnChild(); > >> >> GC.KeepAlive(parent); > >> >> > >> >> will prevent from garbage collecting the parent until the > >> >> OperateOnChild() returns. Try to add GC.KeepAlive() > >> >> at the end of the function. > >> > > >> >Tried that before, no help. Now, I have no clear idea how GC.KeepAlive() > >> >works > >> >on object that is maintained across ASP.NET session. > >> > > >> >> 3. Multithreading issues. > >> >> To properly isolate this issue a global lock should be added when > >> >> invoking the mapscript code (practically at mapscript.dll). I will > >> >> consider how it can be done easily. Until then you can make a test > >> >> against the builtin shape or the posgis driver to see whether the > >> >> problem is SDE related or not. > >> > > >> >I'm still yet to test this on a postgis driver or just the builtin shape > >> >driver. I'll keep you posted on what happens when I do. > >> > > >> >Another thing curious: I'm also using ESRI ArcIMS in my project and the > >> >ActiveX connector seems to run just fine in 64-bit environment, without > >any > >> >special set up. Granted the thing is also a COM object, and COM object > >is > >> >suppported in .NET framework officially. However, I don't feel it is the > >> >same > >> >with pINVOKE, which is responsible for MapServer on .NET. Maybe pINVOKE > >is > >> >buggy under 64-bit of windows. My end goal is to ditch ArcIMS (it has > >its > >> >own > >> >set of problems) completely in my project and have MapServer in its > >place. > >> >I > >> >think I'm getting there, but still there is some way to go. > >> > > >> >I'll keep this thread updated on my progress. > >> > > >> >Thanks. > >> > > >> >Hong > >> > > >> >> 2007/2/1, Hong <[EMAIL PROTECTED]>: > >> >> >An update for my progress: > >> >> > > >> >> >After calling clone() before calling draw() of mapObj, MapServer > >became > >> >> >more > >> >> >stable, but under heavier load (many more dynamic layers and config), > >> >the > >> >> >memory corruption still happened. > >> >> > > >> >> >The strange solution I had found was to compile MapServer using > >Visual > >> >> >Studio > >> >> >2003 instead of 2005. After doing so, MapServer no longer suffered > >> >memory > >> >> >corruption. For some reason, the binary produced from VS2003 running > >> >under > >> >> >.NET 2.0 did not have any memory management problem. This was done > >with > >> >the > >> >> >mapObj still being maintained through session. > >> >> > > >> >> >After a lot of brainstorming over this, I have come to the conjecture > >> >that > >> >> >the > >> >> >garbage collector in the .NET framework might have been the culprit > >of > >> >the > >> >> >memory corruption. Since MapServer internal objects are managed > >> >internally > >> >> >via > >> >> >PInvoke, they are unmanaged as far as the garbage collector is > >> >concerned. > >> >> >Somehow between ASP.NET page postbacks, the garbage collector came in > >> >and > >> >> >invalidated the references unmanaged objects prematurely, as they are > >> >not > >> >> >referenced by any managed object. Now, objects compiled for .NET 1.1 > >(vs > >> >> >2003) > >> >> >are somehow treated differently by the garbage collector of .NET > >2.0. I > >> >> >don't > >> >> >know the exact mechanism. There are ways to control the garbage > >> >collector, > >> >> >however I would rather not explore them. > >> >> > > >> >> >I would like to hear any other thoughts/comments. > >> >> > > >> >> >Hong > >> >> > > >> >> >On Mon, Jan 22, 2007 at 10:39:38PM -0700, Hong wrote: > >> >> >> I have reopened the bug report. > >> >> >> > >> >> >> On Mon, Jan 22, 2007 at 11:12:39PM +0100, Tamas Szekeres wrote: > >> >> >> > 2007/1/22, Hong <[EMAIL PROTECTED]>: > >> >> >> > > > >> >> >> > >Now instead of using the mapObj stored in the session, I > >perform a > >> >> >clone() > >> >> >> > >on the mapObj from the session, and use the clone to perform > >> >draw(). > >> >> >So > >> >> >> > >far, I > >> >> >> > >have not seen any crash after doing this. The only problem I > >> >observed > >> >> >was > >> >> >> > >that > >> >> >> > >the symbol transparencies was messed up after subsequent clone + > >> >draw. > >> >> >> > >Maybe > >> >> >> > >the clone is not perfect, I'm not sure. > >> >> >> > > > >> >> >> > > >> >> >> > That seems strange. You should describe the problem in more > >detail > >> >and > >> >> >> > reopen the following bug then: > >> >> >> > http://mapserver.gis.umn.edu/bugs/show_bug.cgi?id=931 > >> >> >> > > >> >> >> > Please attach the corresponding mapfile, the symbol definition > >file > >> >> >> > and the symbol image to this bug. > >> >> >> > > >> >> >> > Best Regards, > >> >> >> > > >> >> >> > Tamas > >> >> > > >> > > >
