On Thu, Sep 13, 2007 at 08:10:35PM -0300, Alejandro Lozanoff wrote: > Thanks for your explanation and quick response, however with > -Uusemymalloc it segfaults almost when it starts. At least it showed > that the problem comes from that way, probably the mymalloc is worse > than the OpenBSD one. :P
memory use-wise it is not. perhaps you have hit a geniune perl bug then. > We found what appears to be a workaround on awstats. > Changing $tokenquery=$1||''; to $tokenquery='?'; after we traced what > was awstats running when segfaulting. > It's on awstats.pl line 6574 > ----- awstats 6.7 (build 1.892) (c) 2000-2007 Laurent Destailleur ----- > > It runs perfect and we didnt find any problem, but we don't have a clue > as to why that is... We are still looking for a real solution > > Thanks again, > Alejandro. > > mickey wrote: > > On Thu, Sep 13, 2007 at 05:25:32PM -0300, Alejandro Lozanoff wrote: > > > >> Hello list, > >> > > > > re > > > > > >> We recently updated a 3.7 machine running awstat(perl) to parse all our > >> websites logs with the biggest being around 1GB. > >> When parsing the big log it randomly segfaults on 4.1, 3.9 and 3.8, we > >> tried new clean release installs and it still segfaults. On 3.7 it works > >> flawlessly, on 3.8 which has the same perl version as 3.7 (5.8.6) it > >> still segfaults. The problem is completely random but it tends to happen > >> after its been running for a while as it doesnt happen on small logs (or > >> the probability for it to happen on those files is too low ) > >> > >> As the trace below (from the perl.core) shows, it's an out of bounds > >> problem, then we remember about a change on 3.8: > >> > >> "malloc(3) has been rewritten to use the mmap(2) system call, > >> introducing unpredictable allocation addresses and guard pages, which > >> helps in detecting heap based buffer overflows and prevents various > >> types of attacks." > >> > > > > yes. this increases memory fragmentation immensly resulting > > in (practically) less virtual space available for data. > > as an increased penalty (200-300%) for cpu consuption > > on processes w/ lots (20M and more) malloc(3)ed memory... > > as well increased demand for the physical memory that > > on the overcommiting nature of it you perhaps observe. > > a way around it is only to use perl malloc (sbrk-based) > > > > cu > -- paranoic mickey (my employers have changed but, the name has remained)