------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugs.exim.org/show_bug.cgi?id=1362 Summary: JIT Exec Allocator never frees last empty chunk Product: PCRE Version: 8.33 Platform: All OS/Version: All Status: NEW Severity: bug Priority: medium Component: Code AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected] Hi there, actually, I'm not quite sure whether I understand all this in detail. Have a look at the lines 291 to 300 in file sljit/sljitExecAllocator.c /* The whole chunk is free. */ if (SLJIT_UNLIKELY(!free_block->header.prev_size && header->size == 1)) { /* If this block is freed, we still have (allocated_size / 2) free space. */ if (total_size - free_block->size > (allocated_size * 3 / 2)) { total_size -= free_block->size; sljit_remove_free_block(free_block); free_chunk(free_block, free_block->size + sizeof(struct block_header)); } } As the 2nd comment says, that you are trying to keep at least (allocated_size / 2) free memory around for later use. The problem that I see here is, that when the last used block of the last chunk is freed, the memory is actually not given back to the OS. In that case, total_size and free_block->size are equal, whereas allocated_size is zero, which in turn leads to the comparison 0 > 0, which is not true. So, actually, the allocator always keeps the last chunk, no matter how huge it may be. Since there is no garbage collector mechanism in C/C++, the last chunk remains allocated indefinitely. That is great, if new memory is subsequently requested from the allocator, but its a pain, if not. Since there is no pcre[16|32]_really_release_any_unused_jit_code_memory function available, there is no chance to get rid of that chunk during the program's lifetime. At the end of the day, it is likely the OS, that frees the memory when the applications exits. I don't have a good feeling about exiting my application an letting the OS release my memory... Although, this may be ok for you. There are two solutions for that. You could replace the > with a >= operator (or add + 1 to the right side of the test). However, this will make the memory allocator kind of useless, when only one block of memory is in use at a time. The second solution is a memory cleanup function (maybe with a shorter name than that suggested above). That could look something like this: in file sljitExecAllocator.c SLJIT_API_FUNC_ATTRIBUTE void sljit_free_idle_exec(void) { struct block_header *header; struct free_block* free_block; allocator_grab_lock(); free_block = free_blocks; while (free_block) { header = AS_BLOCK_HEADER(free_block, free_block->size); if (SLJIT_UNLIKELY(!free_block->header.prev_size && header->size == 1)) { total_size -= free_block->size; sljit_remove_free_block(free_block); free_chunk(free_block, free_block->size + sizeof(struct block_header)); } free_block = free_block->next; } allocator_release_lock(); } in file pcre_jit_compile.c add this #if defined COMPILE_PCRE8 PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre_jit_release_memory(void) #elif defined COMPILE_PCRE16 PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre16_jit_release_memory(void) #elif defined COMPILE_PCRE32 PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre32_jit_release_memory(void) #endif { sljit_free_idle_exec(); } to the SUPPORT_JIT branch and empty functions to the corresponding #else branch: #if defined COMPILE_PCRE8 PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre_jit_release_memory(void) #elif defined COMPILE_PCRE16 PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre16_jit_release_memory(void) #elif defined COMPILE_PCRE32 PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre32_jit_release_memory(void) #endif { } finally, in file pcre.h, add PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre_jit_release_memory(void); PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre16_jit_release_memory(void); PCRE_EXP_DECL void PCRE_CALL_CONVENTION pcre32_jit_release_memory(void); The functions pcre[16|32]_jit_release_memory will release all chunks, that are completely free. Since calling pcre[16|32]_free_study is still required for each compiled pattern before any of these functions is invoked, there should be only one final chunk left (which should be empty). So, the above implementation with a while loop may be kind of overkill; the loop will likely run one turn only. After all, I can suggest a third solution for that issue: do not use any memory allocator at all. Just call alloc_chunk and free_chunk directly from sljit_malloc_exec and sljit_free_exec, respectively. As the docs say, studying and JIT compiling a pattern is an expensive operation so, for my mind, allocating the needed memory is already the fastest part in that process. Currently, I don't see huge benefits from having a fast memory allocator. Carsten -- Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email -- ## List details at https://lists.exim.org/mailman/listinfo/pcre-dev
