On 24 February 2015 at 13:07, Edward Zimmermann <[email protected]> wrote:
> Answers mixed in... > > > Von: jan i [mailto:[email protected]] > > Gesendet: Montag, 23. Februar 2015 20:25 > > An: [email protected] > > Betreff: Re: Checking malloc success and adding perror() > > > > On 23 February 2015 at 12:47, Edward Zimmermann > > <[email protected]> > > wrote: > > > > > Hi, > > > > > > Been sort-of out the the discussion-- was on vacation last week-- so > > > excuse me, in advance, if I bring up a point already made. > > > > > hi again, nice to hear from you again. > > > > > > > > > > First of all.. Corthinia is supposed to be C++? If so we don't want > > to > > > use malloc. If its plain C, of course, malloc is probably our first > > > choice for memory allocation. > > > > > No DocFormats (the library part) is strictly C99. The application of > > top can be other languages. > > > > Sure. SWIG.... > > > > > > > > > With the issue of x= malloc(y). This gets more complicated. Linux, > > > https://access.redhat.com/documentation/en- > > US/Red_Hat_Enterprise_Linux > > > /6/html/Performance_Tuning_Guide/s-memory-captun.html > > > > > we could, but I do not think we want to inerfere wih kernel parameters. > > > > I was not suggesting that we muck with the kernel params... but need to > accept the behavior. > Android, for example, sets, I think, by standard the value to 1. That > means that malloc will return a pointer even when there is absolutely no > RAM available-- and if the pointer space is depleted (recall most Androids > are 32-bit kernels) unleashes immediately the OOM killer. > ok got you. > > > > > > > > > > > Can we configure the OOM Killer to be a little nicer? Yes but really > > > only process by process.. And when a process gets killed it's done so > > > in a very silent way.. so if someone is "not in the know" to spot the > > > clues.. it can get quite mysterious why some programs stop working... > > > > > > But let us even pretend that we get a NULL... What is the watermark > > to > > > have gotten it? Can we recover? Any buffers we can quickly dispose > > of? > > > Recovery is not that easy! > > > > > You are some steps ahead, right now we replace malloc with xmalloc to > > have a central place for error handling. > > > > I'm arguing that a central place is the wrong place. Android, for example, > won't get there. > Of course it will......when we replace all calls in the library with a call to xmalloc, of course that code will be executed. > > What is wrong with having different allocate functions for the allocation > of different kinds of objects? > complexity, that is not really needed. > What is wrong with having a bit of checking business logic in these > functions? > Code explosion, which is not really needed...and we have seen in the current code, that it is too often forgotten. > Business logic? A function, for example, that wants to create a scratch > buffer might be smart and only create a buffer suitable to the amount of > free RAM around. Another object creator may want to have a pool.. and > another still just call malloc. > When calling malloc is code such as > > if (((Ptr = (t *)malloc(size)) == NULL) { > /* do error handling? */ > } > > Really polluted? > Don't we want to be able to distinguish between recoverable and > non-recoverable errors? > Well maybe, but really malloc errors are practically non-recoverable. > > > > > > > > > > > Should we pretend that we can get a NULL? Of course. It's good > > > programming practice. Should we wrap malloc with an xmalloc for such > > > testing? No. On systems where malloc might return a NULL we should > > > have for different objects alternative strategies for dealing with an > > > allocation failure. A routine, for example, that wants to create a > > > scratch buffer of x length but could work, albeit slower, with less > > we might make smaller. Etc etc. etc. > > > > > of course we need to check for that, try on a Android or IoS system to > > load a huge documents, and you will most surely reach the limit. > > Sure. Lacking virtual memory--- but having a virtual memory arch.... > That is why we use mmap. > > > > > in windows malloc only works well when you allocate in chunks of 4k > > (NTFS size). But both in windows and linux calling both malloc and free > > will cause a context switch. > > The Microsoft Low Fragmentation Heap allocator works really well with sub > 4k objects--- it is limited to chunks under 16k. > > http://illmatics.com/Understanding_the_LFH.pdf > > > > > IoS and Andriod work with preemptive context switching so here it is > > even more expensive. > > > ??? Android is Linux. When we run a native C code it's call libc (BIONIC, > a BSD-derived variant). For malloc they use Doug Lea's malloc. > Yes and no, it is a heavely modified Linux....the good people even decided against using the embedded kernel function, and instead replaced quite a bit with specific android code. > > > > > > > > > > > I'd suggest we keep to malloc and IF NEEDED-- and only if and when > > > needed--- we use a drop-in replacement (and chances are that we'll > > > NEVER need it much less want one). > > > > > Right now we want to replace the malloc calls with xmalloc, so we do > > not need tons of "if NULL" distributed in the code. Replacing malloc is > > a second discussion. > > > > What is wrong with if NULL? > Nothing, but it gives us better control to have it in 1 central function instead of the about 80-90 places where it is (or is missing) right now. > > > > > > > > > Part of the problem is that we might have different "best" approaches > > > for different operating systems. IOS, Android, Linux, BSD, ... making > > "best" > > > not really the "best" goal.. > > > > > Well with my kernel knowledge, all of them benefit from us allocating a > > chunk during startup, eating more if needed, and freeing it all when we > > are finished. Please remember the typical use will be open a docment, > > do something, save a document and stop the application. > > > > This is an old discussion that has waged for decades. Designing software > that does not free memory I think would be a mistake especially when speed > is not the ultimate issue. > > > > > > > > > > Perror? No. Calling directly a function that is intended to write > > to > > > a console is, in general, a bad thing. > > > > > Why do you see that as bad ? I thought it wrote to stderr which can be > > redirected, but anyhow what do you suggest instead. > > Because it is not nice writing to a console. Sure you can redirect stderr > but how do you know if a 3rd party lib that gets used alongside the code > might too think about redirecting stderr to yet another place or even > closing it.. I'm seen all too often things go really wrong. It's just not > best practice-- I'd even call it "bad practice" in a shared library. > Libraries should return error codes and perhaps have a message sub-system-- > or an interface to one. Stderr is perhaps a tad less ugly than stdout.. BUT > we really really should NOT be using either.. > > All good ideas, but we do not have the resource right now to implement e.g. a message subsystem, a call to perror in 1 central function is easier to replace when somebody wants to do it. rgds jan i > > > > > > rgds > > jan i. > > > > > > > > > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Peter Kelly [mailto:[email protected]] > > > Gesendet: Donnerstag, 19. Februar 2015 13:41 > > > An: [email protected] > > > Betreff: Re: Checking malloc success and adding perror() > > > > > > > On 19 Feb 2015, at 7:06 pm, Dennis E. Hamilton > > > > <[email protected]> > > > wrote: > > > > > > > > +1 about a cheap check and common abort procedure for starters. > > > > > > > > I think figuring out what to do about cleanup and exception > > > > unwinding, > > > and even what exception handling to use (if any) is a further > > > platform-development issue that could be masked with simple > > > still-inlineable code, but needs much more architectural thought. > > > > > > I’m fine with us using wrapper functions for these which do the > > checks > > > - though please let’s use xmalloc, xcalloc, xrealloc, and xstrdup > > > instead of > > > DFPlatform* (it’s nothing to do with platform abstraction, and these > > > names are easier to type). (as a side note we can probably cut down > > on > > > prefix usage a lot as long as we don’t export symbols; this was just > > > to avoid name clashes with other libraries) > > > > > > In my previous mail I really just wanted to point out that by itself, > > > this doesn’t really solve anything - the issue is in reality far more > > > complicated than a simple NULL pointer check. > > > > > > I can think of two ways we could deal with the issue of graceful > > handling: > > > > > > 1) Allow the application to supply a callback, as Jan suggested > > > > > > 2) Adopt a “memory pool” type strategy where we create an memory pool > > > object at the start of conversion which tracks all allocations that > > > occur between the beginning and end of a top-level API call like > > > DFGet, and put setjmp/longjmp-style exception handling in these API > > calls. > > > > > > The second approach is in fact already used to a limited extent with > > > the DOM API. Every document maintains its own memory pool for storing > > > Node objects (and the text values of nodes)… this is freed when the > > > document’s retainCount drops to zero. I did this because it was much > > > faster than traversing through the tree and releasing nodes > > > individually (at least in comparison to have nodes as Objective C > > > objects - the ObjC runtime was undoubtedly part of that overhead). > > > > > > — > > > Dr Peter M. Kelly > > > [email protected] > > > > > > PGP key: http://www.kellypmk.net/pgp-key > > > <http://www.kellypmk.net/pgp-key> (fingerprint 5435 6718 59F0 DD1F > > > BFA0 5E46 2523 BAA1 44AE 2966) > > > > > > >
