Re: [Wireshark-dev] New packet list - out of memory?

2009-10-09 Thread Jeff Morriss
Anders Broman wrote: I don't think we can/should turn off canaries in se_ allocations. Instead we should create a new canary-less allocator. (Not sure what such a thing should be named, of course...) Well as I see it EP memory is not a problem we only use one chunk (10M) During the life

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-09 Thread Anders Broman
-Ursprungligt meddelande- Från: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] För Jeff Morriss Skickat: den 9 oktober 2009 22:06 Till: Developer support list for Wireshark Ämne: Re: [Wireshark-dev] New packet list - out of memory? Anders Broman wrote

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread didier
Le mercredi 07 octobre 2009 à 17:01 +0200, Anders Broman a écrit : Hi, If some one's interested here's the emem statistics for the file: #define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64) PWS Commit 43 49264 512 332 052 471 116

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread Anders Broman
oktober 2009 09:34 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] New packet list - out of memory? Le mercredi 07 octobre 2009 à 17:01 +0200, Anders Broman a écrit : Hi, If some one's interested here's the emem statistics for the file: #define EMEM_ALLOCS_PER_CHUNK

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread Jeff Morriss
didier wrote: But are canaries used at all? In my understanding without DEBUG_INTENSE_CANARY_CHECKS they are never checked and it's unset by default. Erm, emem_free_all() checks that the canaries haven't been corrupted: if (memcmp(npc-canary_info-canary[i], canary,

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread Anders Broman
-Original Message- From: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Jeff Morriss Sent: den 8 oktober 2009 15:58 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] New packet list - out of memory? didier wrote

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread Gerald Combs
Anders Broman wrote: Well as I see it EP memory is not a problem we only use one chunk (10M) During the life time of a packet so memory efficency isn't a big issue. But when dealing with large files waisting +30% of the memory is not an option I think. A way to still test se_alloc() could

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread didier
Hi, Le jeudi 08 octobre 2009 à 09:58 -0400, Jeff Morriss a écrit : didier wrote: But are canaries used at all? In my understanding without DEBUG_INTENSE_CANARY_CHECKS they are never checked and it's unset by default. Erm, emem_free_all() checks that the canaries haven't been corrupted:

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread Bryant Eastham
Gerald Combs wrote: We could add a preference item which lets the user disable SE canaries. Maybe a slider with fast on one end and safe on the other, just to let them know what they're getting into. :) Not knowing the extent of the :) above, it is worth pointing out that users are not

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread didier
Hi, Le mercredi 07 octobre 2009 à 13:59 +0200, Anders Broman a écrit : Didier More or less vanilla Wireshark, no colors. Enclosed proto Hierarchy. I can't share this file unfortunatly. The reason why I chose it is just that it caused 'Out of memmory' with the new packet list. Thanks, Are you

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-08 Thread Anders Broman
-Ursprungligt meddelande- Från: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] För didier Skickat: den 9 oktober 2009 03:45 Till: Developer support list for Wireshark Ämne: Re: [Wireshark-dev] New packet list - out of memory? Hi, Le mercredi 07 octobre

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-06 Thread Anders Broman
-Original Message- From: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Bill Meier Sent: den 6 oktober 2009 00:49 To: Developer support list for Wireshark Subject: Re: [Wireshark-dev] New packet list - out of memory? Bill Meier wrote

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-06 Thread Jeff Morriss
Anders Broman wrote: Should we have a new allocator without the canaries? I think either the current se_ allocations implementation need to be changed Or the function removed. The result where WIRESHARK_DEBUG_SE_NO_CHUNKS=T And loading is faster is puzzling. Trying that on Linux (64-bit) with

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-06 Thread didier
Le mardi 06 octobre 2009 à 12:05 +0200, Anders Broman a écrit : 146 MB 661417 pkts (TCP reassembly) Version 1.2.2 (SVN Rev 29910) PWS Commit 42 01661 876 704 656 1 639 256 Loading time ~57s Old packet list SVN 30353 PWS

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-06 Thread didier
Le mardi 06 octobre 2009 à 12:21 -0400, Jeff Morriss a écrit : Another question: do we really need to use se_ allocations here? Do we need the canaries or are we just using se_ allocations because they get cleaned up for us automatically? PacketListRecord size is 20 bytes (could be 16) and

[Wireshark-dev] New packet list - out of memory?

2009-10-05 Thread Anders Broman
Hi, I got a 146 Mbyte trace file, loading it with the new packet list WS blows up Out of memory i in emem.c (~line 353) Looking in the task manager 279 216K s allcated. Using my second build without the new packet list The file loads fine using 492 304K. Changing se_alloc to g_slice_alloc0

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-05 Thread Anders Broman
From: wireshark-dev-boun...@wireshark.org [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of Anders Broman Sent: den 5 oktober 2009 12:32 To: Developer support list for Wireshark Subject: [Wireshark-dev] New packet list - out of memory? Hi, I got a 146 Mbyte trace file, loading

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-05 Thread Bill Meier
Anders Broman wrote: Hi, Experimenting loading the file there is 163 calls to VirtualAlloc() ~1.630 GB memory used Changing #define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 512) to: #define EMEM_ALLOCS_PER_CHUNK (EMEM_PACKET_CHUNK_SIZE / 64) Gives only 26 VirtualAlloc() so it

Re: [Wireshark-dev] New packet list - out of memory?

2009-10-05 Thread Bill Meier
Bill Meier wrote: A data point: A few days ago while working on bug #2375 I noticed that on Windows that if I defined the env logicals to not use se_alloc etc that *much* less memory was required to load a capture. (20% of that used with se_alloc etc ? I don't quite remember). Details: