Re: [Warzone-dev] Memory design problems.
Am Samstag, 16. Juni 2007 schrieb [EMAIL PROTECTED]: On Fri, 15 Jun 2007 14:35:08 -0400 Giel van Schijndel [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] schreef: They do keep track of all objects in treap, and the macro remove of , #define PTRVALID(ptr, size)memPointerValid(ptr, size) did this work for us. Replace PTRVALID() with if(ptr)||==NULL) checks is not same thing what original does! Did you actually check the r990 diff before you pressed the send button in your mail client? http://svn.gna.org/viewcvs/warzone/trunk/lib/framework/mem.h?rev=99 0view=diffr1=990r2=989p1=trunk/lib/framework/mem.hp2=/trunk/li b/framework/mem.h Yes. This is debug builds. How else to find problems? You debug in release mode? How shall debugmode PTRVALID help avoid crashes in release builds? --Dennis signature.asc Description: This is a digitally signed message part. ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On Sat, 16 Jun 2007 08:26:25 -0400 Dennis Schridde [EMAIL PROTECTED] wrote: Am Samstag, 16. Juni 2007 schrieb [EMAIL PROTECTED]: On Fri, 15 Jun 2007 14:35:08 -0400 Giel van Schijndel [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] schreef: They do keep track of all objects in treap, and the macro remove of , #define PTRVALID(ptr, size) memPointerValid(ptr, size) did this work for us. Replace PTRVALID() with if(ptr)||==NULL) checks is not same thing what original does! Did you actually check the r990 diff before you pressed the send button in your mail client? http://svn.gna.org/viewcvs/warzone/trunk/lib/framework/mem.h?rev=9 9 0view=diffr1=990r2=989p1=trunk/lib/framework/mem.hp2=/trunk/l i b/framework/mem.h Yes. This is debug builds. How else to find problems? You debug in release mode? How shall debugmode PTRVALID help avoid crashes in release builds? They will no help in release build, they not meant to. This why I say debug builds it help. I am trying to find memory bugs in debug builds, since we know what values are for wrong pointers. If find errors in debug builds, then you will no see crash in release builds. They have uninit, dangling, and corruption checks. -- Click to find great rates on home insurance, save big, shop here http://tagline.hushmail.com/fc/CAaCXv1QU9DHxazsqCXddTzH9qLpok5U/ ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On 6/16/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: They will no help in release build, they not meant to. This why I say debug builds it help. I am trying to find memory bugs in debug builds, since we know what values are for wrong pointers. If find errors in debug builds, then you will no see crash in release builds. They have uninit, dangling, and corruption checks. What we are trying to tell you earlier in this thread is that the original memory management system in Warzone was hiding bugs that we have now unmasked, and are able to remove. It also allows us to understand more clearly what is going on in the code, and debug it using modern tools such as valgrind. BTW, come to think of it, we can add a more thorough check for dangling pointers in the CHECK_*() macros if we want to, by checking if an object's pointer is in the object lists. - Per ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On Wed, 13 Jun 2007 18:13:44 -0400 Giel van Schijndel [EMAIL PROTECTED] wrote: Dennis Schridde schreef: There must have been some special assumption on Pumpkins heap implementation which magically prevented eternal doom. Till now no one found out how it worked that wonder. I've got a very good clue on how that heap thingy worked in preventing segmentation faults. Since a HEAP_FREE never actually released the memory back to the OS or to the malloc/free implementation we simply where never punished (by segfaults) if we _did_ access dangling pointers anyway. Now that we do use mallocfree however memory is much more volatile (since all memory is now allocated from one pool rather than several dedicated pools), which makes memory to which dangling pointers point more likely to change in value, and thus become invalid to whatever function uses it. Keep in mind though that with both the heap system as well as malloc/free the pointers where _wrong_, it's just that now we are able to actually debug them, and feel the pain resulting from lots of years where debugging wasn't performed. I no think you right. It was correct, it was designs this way. In rev 990 muggenhor, removes checks for invalid pointers. Why? They do keep track of all objects in treap, and the macro remove of , #define PTRVALID(ptr, size) memPointerValid(ptr, size) did this work for us. Replace PTRVALID() with if(ptr)||==NULL) checks is not same thing what original does! Original will call memPointerValid(ptr, size). This then check if the block is in the treap. If yes, then IS valid pointer/object. If no, then object is NO valid, returns FALSE, calling routine returns back without doing anything. Now Crash in our cases, since pointer/object is NO NULL, which is the new check, so this is wrong. I debug full missions that crash on current (see bugs reports), and compare back to berlios versions. I then set breakpoints at code where current crash, and check pointers. I see 'invalids' pointers, and game deals with them as program to. This why I say just because you no like this design of original, no make it wrong? It worked for them fine did it not? -- Click for special offer on replacement windows - energy efficient http://tagline.hushmail.com/fc/CAaCXv1SRWqeB3jZeDH7xTcLPgTdhJYj/ ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
[EMAIL PROTECTED] schreef: On Wed, 13 Jun 2007 18:13:44 -0400 Giel van Schijndel [EMAIL PROTECTED] wrote: Dennis Schridde schreef: There must have been some special assumption on Pumpkins heap implementation which magically prevented eternal doom. Till now no one found out how it worked that wonder. I've got a very good clue on how that heap thingy worked in preventing segmentation faults. Since a HEAP_FREE never actually released the memory back to the OS or to the malloc/free implementation we simply where never punished (by segfaults) if we _did_ access dangling pointers anyway. Now that we do use mallocfree however memory is much more volatile (since all memory is now allocated from one pool rather than several dedicated pools), which makes memory to which dangling pointers point more likely to change in value, and thus become invalid to whatever function uses it. Keep in mind though that with both the heap system as well as malloc/free the pointers where _wrong_, it's just that now we are able to actually debug them, and feel the pain resulting from lots of years where debugging wasn't performed. I no think you right. It was correct, it was designs this way. In rev 990 muggenhor, removes checks for invalid pointers. Why? They do keep track of all objects in treap, and the macro remove of , #define PTRVALID(ptr, size) memPointerValid(ptr, size) did this work for us. Replace PTRVALID() with if(ptr)||==NULL) checks is not same thing what original does! Did you actually check the r990 diff before you pressed the send button in your mail client? http://svn.gna.org/viewcvs/warzone/trunk/lib/framework/mem.h?rev=990view=diffr1=990r2=989p1=trunk/lib/framework/mem.hp2=/trunk/lib/framework/mem.h The PTRVALID macro was ptr != NULL in r985:989. See this: #define PTRVALID(ptr, size) (ptr != NULL) So all I did there was expanding the PTRVALID macro (to increase readability). Then you might be interested to see this diff (r985): http://svn.gna.org/viewcvs/warzone/trunk/lib/framework/mem.h?rev=985view=diffr1=985r2=984p1=trunk/lib/framework/mem.hp2=/trunk/lib/framework/mem.h With emphasis on this piece of the original code: #ifdef DEBUG_MALLOC # ifndef NO_PTRVALID # define PTRVALID(ptr, size) memPointerValid(ptr, size) # else # define PTRVALID(ptr, size) (((ptr)==NULL)?FALSE:TRUE) # endif #else // !DEBUG_MALLOC # define PTRVALID(ptr, size) (TRUE) #endif // DEBUG_MALLOC As you can see, this PTRVALID macro only actually did anything when compiled in debugging mode. Therefore saying it was correctly designed and that removing the PTRVALID macro invalidated the correct system is wrong. Original will call memPointerValid(ptr, size). This then check if the block is in the treap. If yes, then IS valid pointer/object. If no, then object is NO valid, returns FALSE, calling routine returns back without doing anything. Now Crash in our cases, since pointer/object is NO NULL, which is the new check, so this is wrong. I debug full missions that crash on current (see bugs reports), and compare back to berlios versions. I then set breakpoints at code where current crash, and check pointers. I see 'invalids' pointers, and game deals with them as program to. This why I say just because you no like this design of original, no make it wrong? It worked for them fine did it not? No I don't like how the memory management for Warzone is designed. And sticking with this because it worked for them is a fallacy (look it, i.e. fallacy, up on wikipedia if you like). -- Giel signature.asc Description: OpenPGP digital signature ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On Fri, 15 Jun 2007 14:35:08 -0400 Giel van Schijndel [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] schreef: They do keep track of all objects in treap, and the macro remove of , #define PTRVALID(ptr, size) memPointerValid(ptr, size) did this work for us. Replace PTRVALID() with if(ptr)||==NULL) checks is not same thing what original does! Did you actually check the r990 diff before you pressed the send button in your mail client? http://svn.gna.org/viewcvs/warzone/trunk/lib/framework/mem.h?rev=99 0view=diffr1=990r2=989p1=trunk/lib/framework/mem.hp2=/trunk/li b/framework/mem.h Yes. This is debug builds. How else to find problems? You debug in release mode? The PTRVALID macro was ptr != NULL in r985:989. See this: #define PTRVALID(ptr, size) (ptr != NULL) So all I did there was expanding the PTRVALID macro (to increase readability). As you can see, this PTRVALID macro only actually did anything when compiled in debugging mode. Therefore saying it was correctly designed and that removing the PTRVALID macro invalidated the correct system is wrong. ? We try to find dangling pointers, and other memory issues. They have debug system in code already just for these errors. So it was correct code they have for debug builds to find problems. just because you no like this design of original, no make it wrong? It worked for them fine did it not? No I don't like how the memory management for Warzone is designed. And sticking with this because it worked for them is a fallacy (look it, i.e. fallacy, up on wikipedia if you like). Fallacy = flaw ? I am no say stick with it. I am say that we insert it back in to help debug problems we have now, so to see where things go wrong. The checks they had did catch most errors for them, now we need same help to detect these errors. As I say before, efence/valgrind no really help, unless you know better how to use them? If yes, share with rest so we can fix this? -- Click to find great rates on home insurance, save big, shop here http://tagline.hushmail.com/fc/CAaCXv1QU9GHggbX8JSWf6sOy12HX4kv/ ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On 6/13/07, Giel van Schijndel [EMAIL PROTECTED] wrote: Dennis Schridde schreef: - Refcounting is probably even better. I like reference counting, in fact it is to my opinion both the safest and most efficient solution of all. However, I do think that unlike C++, which provides constructors and destructors, reference counting will not be easy to use. This is because before destroying a pointer (either because it leaves scope, or whatever other reason) you'd have to call a special destruction function (which decrements the refcount and if it reaches zero destroys the pointed to object in the appropriate way), each and every time a pointer becomes unused. Similarly each and every time you'd copy a pointer you'd have to make sure the reference count is increased. This may seem trivial, but applying this technique on every single pointer we've got is most certainly _not_ trivial. On the bright side, you do not have to do it for *all* pointers, only global pointers, and local pointers in very special functions where the references of your pointers may disappear *during* local execution. - Per ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
Am Mittwoch, 13. Juni 2007 schrieb Giel van Schijndel: Dennis Schridde schreef: - Refcounting is probably even better. I like reference counting, in fact it is to my opinion both the safest and most efficient solution of all. However, I do think that unlike C++, which provides constructors and destructors, reference counting will not be easy to use. This is because before destroying a pointer (either because it leaves scope, or whatever other reason) you'd have to call a special destruction function (which decrements the refcount and if it reaches zero destroys the pointed to object in the appropriate way), each and every time a pointer becomes unused. Similarly each and every time you'd copy a pointer you'd have to make sure the reference count is increased. This may seem trivial, but applying this technique on every single pointer we've got is most certainly _not_ trivial. Most prominent user of these smart-pointers would be the droid/projectile lists, where refcounting could probably easily be deployed. --Dennis signature.asc Description: This is a digitally signed message part. ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On 6/13/07, Dennis Schridde [EMAIL PROTECTED] wrote: - Dead-lists (prune-lists, as they are called in WZ according to Per) No, that's what I called them, because that is what I named a somewhat similar system that I made once. It is called list of destroyed objects or similar in Warzone. Just to nitpick on an otherwise fine post ;-) - Per ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On Wed, 13 Jun 2007 14:16:26 -0400 Dennis Schridde [EMAIL PROTECTED] wrote: Am Mittwoch, 13. Juni 2007 schrieb [EMAIL PROTECTED]: On Mon, 11 Jun 2007 16:45:58 -0400 Dennis Schridde Valgrind should help... Will try on the weekend. You apparently dove deep into the code and found several places where unitialised memory comes from or pointers are not set to NULL. Maybe you can create a list? (variablename, file, line) Create list of all variables that crash on? It looks to me this is most for droids. (target/sound/and so ) I think, could be best is when droid is made, put this into global (linked list?) list. Then on droid dead, we clear entry in list. Then before all calls that have to use droid data, we check if match on global list. If yes, then continue. If no, then abort out of routine.This list may be, 300 units (max?) for each player. I thinks this will works OK. Maybe have to do this for all other types units/buildings also? I am not sure whether I understood you correctly, in case you want an alive-list with all droids being alive and before using any pointer, walk the list to see whether it is in that list, is maybe a good idea, and maybe not. Our thoughts about this on IRC were as follows: - Walking a long list with all units in it is slooow. Walking this list before dereferencing any pointer is possibly even slower. - Dead-lists (prune-lists, as they are called in WZ according to Per) are probably better to walk, because they usually are shorter. - Refcounting is probably even better. - Fully ID based system is another option, but maybe too slow. It could be speed up by caching the pointer in case you can ensure that you don't destroy the droid while the function (...) runs. This needs some work of the user (of the ID system), though. Another option is use heap again. I am no sure what is best. If we walk list, 300 is not too bad list to walk. Max players is 8, so 2400 is max list size. (live/about to die, and no in list, then it is dead) How does bullets/missiles work in game? Does it keep track in list someplace? I will try see more info how berlios version works. -- Click for a free comparison on healthcare coverage and save 100's http://tagline.hushmail.com/fc/CAaCXv1QUc0Q43ytBsnmKZueOrv7gaae/ ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
Am Mittwoch, 13. Juni 2007 schrieb [EMAIL PROTECTED]: On Wed, 13 Jun 2007 14:16:26 -0400 Dennis Schridde [EMAIL PROTECTED] wrote: Am Mittwoch, 13. Juni 2007 schrieb [EMAIL PROTECTED]: On Mon, 11 Jun 2007 16:45:58 -0400 Dennis Schridde Valgrind should help... Will try on the weekend. You apparently dove deep into the code and found several places where unitialised memory comes from or pointers are not set to NULL. Maybe you can create a list? (variablename, file, line) Create list of all variables that crash on? It looks to me this is most for droids. (target/sound/and so ) I think, could be best is when droid is made, put this into global (linked list?) list. Then on droid dead, we clear entry in list. Then before all calls that have to use droid data, we check if match on global list. If yes, then continue. If no, then abort out of routine.This list may be, 300 units (max?) for each player. I thinks this will works OK. Maybe have to do this for all other types units/buildings also? I am not sure whether I understood you correctly, in case you want an alive-list with all droids being alive and before using any pointer, walk the list to see whether it is in that list, is maybe a good idea, and maybe not. Our thoughts about this on IRC were as follows: - Walking a long list with all units in it is slooow. Walking this list before dereferencing any pointer is possibly even slower. - Dead-lists (prune-lists, as they are called in WZ according to Per) are probably better to walk, because they usually are shorter. - Refcounting is probably even better. - Fully ID based system is another option, but maybe too slow. It could be speed up by caching the pointer in case you can ensure that you don't destroy the droid while the function (...) runs. This needs some work of the user (of the ID system), though. Another option is use heap again. The magic question is: How is that going to improve the situation? There must have been some special assumption on Pumpkins heap implementation which magically prevented eternal doom. Till now no one found out how it worked that wonder. --Dennis signature.asc Description: This is a digitally signed message part. ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
Dennis Schridde schreef: Am Mittwoch, 13. Juni 2007 schrieb [EMAIL PROTECTED]: Another option is use heap again. The magic question is: How is that going to improve the situation? There must have been some special assumption on Pumpkins heap implementation which magically prevented eternal doom. Till now no one found out how it worked that wonder. I've got a very good clue on how that heap thingy worked in preventing segmentation faults. Since a HEAP_FREE never actually released the memory back to the OS or to the malloc/free implementation we simply where never punished (by segfaults) if we _did_ access dangling pointers anyway. Now that we do use mallocfree however memory is much more volatile (since all memory is now allocated from one pool rather than several dedicated pools), which makes memory to which dangling pointers point more likely to change in value, and thus become invalid to whatever function uses it. Keep in mind though that with both the heap system as well as malloc/free the pointers where _wrong_, it's just that now we are able to actually debug them, and feel the pain resulting from lots of years where debugging wasn't performed. -- Giel signature.asc Description: OpenPGP digital signature ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
On Mon, 11 Jun 2007 16:45:58 -0400 Dennis Schridde [EMAIL PROTECTED] wrote: Am Montag, 11. Juni 2007 schrieb [EMAIL PROTECTED]: I now trace back many problems now with warzone. It all because of switch from MALLOC/FREE, and HEAP usage to malloc/free, as I try to say before. I finds that most all crashes I find is because of this, and also most all crashes in games both SP and MP is because of this changes. The game use HEAP, and would always do memset to clear out everything. The game was design this way. I thinks PS2 is main reasons for this, to save many calls, and footprint of code size. With change to malloc/free no HEAP, then we have dangling pointers (0x) use. We also have many uninit (0xcdcdcdcd) use. From svn 1100, reverts back to this. Now do as in bug 9235, and 9233 no happen. Others bugs also no happen. Then update to next version, and now happens all time you try. Way to part fix is to go through all code and do as I say before. We still have problems of copy of structures, since one area may be clear and fall out of scope, but copy still here, and when try to access, it crash. This what seem to happen in 9235 9233, and others. When game before use HEAP, when clear everything in HEAP was clear to NULL. This how they reset game elements for many things. Very hard to find memory bugs now. We gots stack trace, but this no show where original 0x/0xcdcdcdcd happens. Is any tools to help this? I no have $1300 for Rational PurifyPlus, which is mades to find this. Anything on linux? I try efence/valgrind, and it no help really. Valgrind should help... Will try on the weekend. You apparently dove deep into the code and found several places where unitialised memory comes from or pointers are not set to NULL. Maybe you can create a list? (variablename, file, line) Create list of all variables that crash on? It looks to me this is most for droids. (target/sound/and so ) I think, could be best is when droid is made, put this into global (linked list?) list. Then on droid dead, we clear entry in list. Then before all calls that have to use droid data, we check if match on global list. If yes, then continue. If no, then abort out of routine.This list may be, 300 units (max?) for each player. I thinks this will works OK. Maybe have to do this for all other types units/buildings also? -- Patio furniture that can last you a lifetime. Below retail. Click Now. http://tagline.hushmail.com/fc/CAaCXv1UT8WDmne20hDyZOspBR1KPqQv/ ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
Re: [Warzone-dev] Memory design problems.
Am Montag, 11. Juni 2007 schrieb [EMAIL PROTECTED]: I now trace back many problems now with warzone. It all because of switch from MALLOC/FREE, and HEAP usage to malloc/free, as I try to say before. I finds that most all crashes I find is because of this, and also most all crashes in games both SP and MP is because of this changes. The game use HEAP, and would always do memset to clear out everything. The game was design this way. I thinks PS2 is main reasons for this, to save many calls, and footprint of code size. With change to malloc/free no HEAP, then we have dangling pointers (0x) use. We also have many uninit (0xcdcdcdcd) use. From svn 1100, reverts back to this. Now do as in bug 9235, and 9233 no happen. Others bugs also no happen. Then update to next version, and now happens all time you try. Way to part fix is to go through all code and do as I say before. We still have problems of copy of structures, since one area may be clear and fall out of scope, but copy still here, and when try to access, it crash. This what seem to happen in 9235 9233, and others. When game before use HEAP, when clear everything in HEAP was clear to NULL. This how they reset game elements for many things. Very hard to find memory bugs now. We gots stack trace, but this no show where original 0x/0xcdcdcdcd happens. Is any tools to help this? I no have $1300 for Rational PurifyPlus, which is mades to find this. Anything on linux? I try efence/valgrind, and it no help really. Valgrind should help... Will try on the weekend. You apparently dove deep into the code and found several places where unitialised memory comes from or pointers are not set to NULL. Maybe you can create a list? (variablename, file, line) Thanks, Dennis signature.asc Description: This is a digitally signed message part. ___ Warzone-dev mailing list Warzone-dev@gna.org https://mail.gna.org/listinfo/warzone-dev
