Just wondered if you read the rest of the thread. The intention is to replace perror() with a call to an application defined function pointer, that way different application can do different handling.
rgds jan I. On 24 February 2015 at 14:03, jan i <[email protected]> wrote: > > > > On 24 February 2015 at 13:07, Edward Zimmermann <[email protected]> > wrote: > >> Answers mixed in... >> >> > Von: jan i [mailto:[email protected]] >> > Gesendet: Montag, 23. Februar 2015 20:25 >> > An: [email protected] >> > Betreff: Re: Checking malloc success and adding perror() >> > >> > On 23 February 2015 at 12:47, Edward Zimmermann >> > <[email protected]> >> > wrote: >> > >> > > Hi, >> > > >> > > Been sort-of out the the discussion-- was on vacation last week-- so >> > > excuse me, in advance, if I bring up a point already made. >> > > >> > hi again, nice to hear from you again. >> > >> > >> > > >> > > First of all.. Corthinia is supposed to be C++? If so we don't want >> > to >> > > use malloc. If its plain C, of course, malloc is probably our first >> > > choice for memory allocation. >> > > >> > No DocFormats (the library part) is strictly C99. The application of >> > top can be other languages. >> > >> >> Sure. SWIG.... >> >> > >> > > >> > > With the issue of x= malloc(y). This gets more complicated. Linux, >> > > https://access.redhat.com/documentation/en- >> > US/Red_Hat_Enterprise_Linux >> > > /6/html/Performance_Tuning_Guide/s-memory-captun.html >> > > >> > we could, but I do not think we want to inerfere wih kernel parameters. >> > >> >> I was not suggesting that we muck with the kernel params... but need to >> accept the behavior. >> Android, for example, sets, I think, by standard the value to 1. That >> means that malloc will return a pointer even when there is absolutely no >> RAM available-- and if the pointer space is depleted (recall most Androids >> are 32-bit kernels) unleashes immediately the OOM killer. >> > > ok got you. > >> >> >> >> > >> > > >> > > Can we configure the OOM Killer to be a little nicer? Yes but really >> > > only process by process.. And when a process gets killed it's done so >> > > in a very silent way.. so if someone is "not in the know" to spot the >> > > clues.. it can get quite mysterious why some programs stop working... >> > > >> > > But let us even pretend that we get a NULL... What is the watermark >> > to >> > > have gotten it? Can we recover? Any buffers we can quickly dispose >> > of? >> > > Recovery is not that easy! >> > > >> > You are some steps ahead, right now we replace malloc with xmalloc to >> > have a central place for error handling. >> > >> >> I'm arguing that a central place is the wrong place. Android, for >> example, won't get there. >> > Of course it will......when we replace all calls in the library with a > call to xmalloc, of course that code will be executed. > > >> >> What is wrong with having different allocate functions for the allocation >> of different kinds of objects? >> > complexity, that is not really needed. > > >> What is wrong with having a bit of checking business logic in these >> functions? >> > Code explosion, which is not really needed...and we have seen in the > current code, that it is too often forgotten. > > >> Business logic? A function, for example, that wants to create a scratch >> buffer might be smart and only create a buffer suitable to the amount of >> free RAM around. Another object creator may want to have a pool.. and >> another still just call malloc. >> When calling malloc is code such as >> >> if (((Ptr = (t *)malloc(size)) == NULL) { >> /* do error handling? */ >> } >> >> Really polluted? >> Don't we want to be able to distinguish between recoverable and >> non-recoverable errors? >> > Well maybe, but really malloc errors are practically non-recoverable. > > >> >> >> >> > >> > > >> > > Should we pretend that we can get a NULL? Of course. It's good >> > > programming practice. Should we wrap malloc with an xmalloc for such >> > > testing? No. On systems where malloc might return a NULL we should >> > > have for different objects alternative strategies for dealing with an >> > > allocation failure. A routine, for example, that wants to create a >> > > scratch buffer of x length but could work, albeit slower, with less >> > we might make smaller. Etc etc. etc. >> > > >> > of course we need to check for that, try on a Android or IoS system to >> > load a huge documents, and you will most surely reach the limit. >> >> Sure. Lacking virtual memory--- but having a virtual memory arch.... >> That is why we use mmap. >> >> > >> > in windows malloc only works well when you allocate in chunks of 4k >> > (NTFS size). But both in windows and linux calling both malloc and free >> > will cause a context switch. >> >> The Microsoft Low Fragmentation Heap allocator works really well with sub >> 4k objects--- it is limited to chunks under 16k. >> >> http://illmatics.com/Understanding_the_LFH.pdf >> >> > >> > IoS and Andriod work with preemptive context switching so here it is >> > even more expensive. >> > >> ??? Android is Linux. When we run a native C code it's call libc (BIONIC, >> a BSD-derived variant). For malloc they use Doug Lea's malloc. >> > > Yes and no, it is a heavely modified Linux....the good people even decided > against using the embedded kernel function, and instead replaced quite a > bit with specific android code. > > >> >> >> >> > >> > > >> > > I'd suggest we keep to malloc and IF NEEDED-- and only if and when >> > > needed--- we use a drop-in replacement (and chances are that we'll >> > > NEVER need it much less want one). >> > > >> > Right now we want to replace the malloc calls with xmalloc, so we do >> > not need tons of "if NULL" distributed in the code. Replacing malloc is >> > a second discussion. >> > >> >> What is wrong with if NULL? >> > > Nothing, but it gives us better control to have it in 1 central function > instead of the about 80-90 places where it is (or is missing) right now. > > >> >> > >> > > >> > > Part of the problem is that we might have different "best" approaches >> > > for different operating systems. IOS, Android, Linux, BSD, ... making >> > "best" >> > > not really the "best" goal.. >> > > >> > Well with my kernel knowledge, all of them benefit from us allocating a >> > chunk during startup, eating more if needed, and freeing it all when we >> > are finished. Please remember the typical use will be open a docment, >> > do something, save a document and stop the application. >> > >> >> This is an old discussion that has waged for decades. Designing software >> that does not free memory I think would be a mistake especially when speed >> is not the ultimate issue. >> >> >> > >> > > >> > > Perror? No. Calling directly a function that is intended to write >> > to >> > > a console is, in general, a bad thing. >> > > >> > Why do you see that as bad ? I thought it wrote to stderr which can be >> > redirected, but anyhow what do you suggest instead. >> >> Because it is not nice writing to a console. Sure you can redirect stderr >> but how do you know if a 3rd party lib that gets used alongside the code >> might too think about redirecting stderr to yet another place or even >> closing it.. I'm seen all too often things go really wrong. It's just not >> best practice-- I'd even call it "bad practice" in a shared library. >> Libraries should return error codes and perhaps have a message sub-system-- >> or an interface to one. Stderr is perhaps a tad less ugly than stdout.. BUT >> we really really should NOT be using either.. >> >> > All good ideas, but we do not have the resource right now to implement > e.g. a message subsystem, a call to perror in 1 central function is easier > to replace when somebody wants to do it. > > rgds > jan i > >> >> >> > >> > rgds >> > jan i. >> > >> > > >> > > >> > > >> > > -----Ursprüngliche Nachricht----- >> > > Von: Peter Kelly [mailto:[email protected]] >> > > Gesendet: Donnerstag, 19. Februar 2015 13:41 >> > > An: [email protected] >> > > Betreff: Re: Checking malloc success and adding perror() >> > > >> > > > On 19 Feb 2015, at 7:06 pm, Dennis E. Hamilton >> > > > <[email protected]> >> > > wrote: >> > > > >> > > > +1 about a cheap check and common abort procedure for starters. >> > > > >> > > > I think figuring out what to do about cleanup and exception >> > > > unwinding, >> > > and even what exception handling to use (if any) is a further >> > > platform-development issue that could be masked with simple >> > > still-inlineable code, but needs much more architectural thought. >> > > >> > > I’m fine with us using wrapper functions for these which do the >> > checks >> > > - though please let’s use xmalloc, xcalloc, xrealloc, and xstrdup >> > > instead of >> > > DFPlatform* (it’s nothing to do with platform abstraction, and these >> > > names are easier to type). (as a side note we can probably cut down >> > on >> > > prefix usage a lot as long as we don’t export symbols; this was just >> > > to avoid name clashes with other libraries) >> > > >> > > In my previous mail I really just wanted to point out that by itself, >> > > this doesn’t really solve anything - the issue is in reality far more >> > > complicated than a simple NULL pointer check. >> > > >> > > I can think of two ways we could deal with the issue of graceful >> > handling: >> > > >> > > 1) Allow the application to supply a callback, as Jan suggested >> > > >> > > 2) Adopt a “memory pool” type strategy where we create an memory pool >> > > object at the start of conversion which tracks all allocations that >> > > occur between the beginning and end of a top-level API call like >> > > DFGet, and put setjmp/longjmp-style exception handling in these API >> > calls. >> > > >> > > The second approach is in fact already used to a limited extent with >> > > the DOM API. Every document maintains its own memory pool for storing >> > > Node objects (and the text values of nodes)… this is freed when the >> > > document’s retainCount drops to zero. I did this because it was much >> > > faster than traversing through the tree and releasing nodes >> > > individually (at least in comparison to have nodes as Objective C >> > > objects - the ObjC runtime was undoubtedly part of that overhead). >> > > >> > > — >> > > Dr Peter M. Kelly >> > > [email protected] >> > > >> > > PGP key: http://www.kellypmk.net/pgp-key >> > > <http://www.kellypmk.net/pgp-key> (fingerprint 5435 6718 59F0 DD1F >> > > BFA0 5E46 2523 BAA1 44AE 2966) >> > > >> > > >> > >
