Re: Memory leak when catting(/sedding/...) large binary files with backticks
Chet Ramey wrote: [EMAIL PROTECTED] wrote: Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -D_GNU_SOURCE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic uname output: Linux pmpc983.npm.ac.uk 2.6.24.4-64.fc8 #1 SMP Sat Mar 29 09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux Machine Type: x86_64-redhat-linux-gnu Bash Version: 3.2 Patch Level: 33 Release Status: release Description: Using echo `cat ...` on a large binary file causes lots of memory to be used (fine), but if you ctrl-c while it's running it doesn't die properly and doesn't return used memory when finished. Originally found by screwing up a sed command (can also reproduce bug using sed rather than cat) while trying to rename a group of files. Repeat-By: Every time 1. Find large binary data file for test (mine is ~3.2GB) 2. echo `cat filename` 3. Ctrl-C previous command while running (doesn't terminate) 4. When step 2 eventually returns it does not release memory I'm not sure what you mean by `doesn't return used memory', but if you mean a process's size as reported by ps or similar, that does not indicate a memory leak. A memory leak is memory that has been allocated by a program to which it retains no handles. malloc acts as a cache between an application and the kernel. Memory obtained from the kernel using malloc may, under some circumstances, be returned to the kernel upon free, but this may not always be possible. Memory that is not returned to the kernel by freeing pages or using sbrk with a negative argument is retained and used to satisfy future requests. I ran your test using valgrind to check for memory leaks (but with only a 330 MB file), and it reported no leaks after ^C. Chet I was just going on what ps reported, but I assumed it was leaking on the basis that the memory did not report as free until I kill -9'd the relevant bash process (just kill didn't work). Once it'd been done a couple of times so most of the memory was consumed, it definitely had an adverse effect on performance - even other simple bash commands took several seconds to return a result, which I assume was because they were fighting for memory. The affected bash also didn't show any sub-processes using ps -auxf (shouldn't it have shown a cat process if it was still holding resources?). If you guys on here reckon it's not a bug that's fine - I admit I'm not exactly au fait with the inner workings of bash, maybe it's working as intended. I just figured since it was eating my memory and not making that memory available to other programs when it was ^C'd (as you would do when you realised you'd inadvertently catted or sedded a 3GB binary file) that I'd report it. Ben
Re: Memory leak when catting(/sedding/...) large binary files with backticks
[EMAIL PROTECTED] wrote: Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -D_GNU_SOURCE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic uname output: Linux pmpc983.npm.ac.uk 2.6.24.4-64.fc8 #1 SMP Sat Mar 29 09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux Machine Type: x86_64-redhat-linux-gnu Bash Version: 3.2 Patch Level: 33 Release Status: release Description: Using echo `cat ...` on a large binary file causes lots of memory to be used (fine), but if you ctrl-c while it's running it doesn't die properly and doesn't return used memory when finished. Originally found by screwing up a sed command (can also reproduce bug using sed rather than cat) while trying to rename a group of files. Repeat-By: Every time 1. Find large binary data file for test (mine is ~3.2GB) 2. echo `cat filename` 3. Ctrl-C previous command while running (doesn't terminate) 4. When step 2 eventually returns it does not release memory I'm not sure what you mean by `doesn't return used memory', but if you mean a process's size as reported by ps or similar, that does not indicate a memory leak. A memory leak is memory that has been allocated by a program to which it retains no handles. malloc acts as a cache between an application and the kernel. Memory obtained from the kernel using malloc may, under some circumstances, be returned to the kernel upon free, but this may not always be possible. Memory that is not returned to the kernel by freeing pages or using sbrk with a negative argument is retained and used to satisfy future requests. I ran your test using valgrind to check for memory leaks (but with only a 330 MB file), and it reported no leaks after ^C. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer Live Strong. No day but today. Chet Ramey, ITS, CWRU[EMAIL PROTECTED]http://cnswww.cns.cwru.edu/~chet/
Memory leak when catting(/sedding/...) large binary files with backticks
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -D_GNU_SOURCE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic uname output: Linux pmpc983.npm.ac.uk 2.6.24.4-64.fc8 #1 SMP Sat Mar 29 09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux Machine Type: x86_64-redhat-linux-gnu Bash Version: 3.2 Patch Level: 33 Release Status: release Description: Using echo `cat ...` on a large binary file causes lots of memory to be used (fine), but if you ctrl-c while it's running it doesn't die properly and doesn't return used memory when finished. Originally found by screwing up a sed command (can also reproduce bug using sed rather than cat) while trying to rename a group of files. Repeat-By: Every time 1. Find large binary data file for test (mine is ~3.2GB) 2. echo `cat filename` 3. Ctrl-C previous command while running (doesn't terminate) 4. When step 2 eventually returns it does not release memory
