Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-04-06 Thread Matthew Woehlke
Chet Ramey wrote: I'm sure there are efficiency improvements possible in the bash indexed array implementation, but sequentially accessing a data structure optimized for space and sparse arrays is never going to be as fast as a read-process loop, and that difference becomes more and more

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-04-06 Thread Chet Ramey
Matthew Woehlke wrote: Chet Ramey wrote: I'm sure there are efficiency improvements possible in the bash indexed array implementation, but sequentially accessing a data structure optimized for space and sparse arrays is never going to be as fast as a read-process loop, and that difference

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-28 Thread Lennart Schultz
It seems that mapfile is OK for small numbers but for bigger numbers it starts to compsume time. I made a litle test: rm Xyz; unset MAPFILE # clear max= # set limit time for i in $(seq 0 $max); do echo 'Xyz' Xyz; done real0m0.490s user0m0.304s sys 0m0.124s time mapfile Xyz

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-28 Thread Chet Ramey
Lennart Schultz wrote: It seems that mapfile is OK for small numbers but for bigger numbers it starts to compsume time. Not exactly. Your own timing tests show that mapfile itself is blindingly fast. The time is consumed sequentially traversing the (very large) array. Bash indexed arrays are

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-27 Thread Stephane CHAZELAS
2009-03-26, 21:22(-04), Chet Ramey: Chris F.A. Johnson wrote: Chet, how about an option to mapfile that strips leading and/or trailing spaces? Another useful option would be to remove newlines. I'm disinclined to add one, since it's easy enough to use the ${line##[ ]} and

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-27 Thread Greg Wooledge
On Thu, Mar 26, 2009 at 05:59:14PM -0400, Chris F.A. Johnson wrote: Chet, how about an option to mapfile that strips leading and/or trailing spaces? Another useful option would be to remove newlines. It already has the latter: -tRemove a trailing newline from

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-27 Thread Lennart Schultz
Chris, I agree with you to use the right tool at the right time, and mapfile seems not to be the right tool for my problem, but I will just give you some facts of my observations: using a fast tool like egrep just to find a simple string in my datafile gives the following times: time egrep 'pro'

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-27 Thread Chris F.A. Johnson
On Fri, 27 Mar 2009, Lennart Schultz wrote: Chris, I agree with you to use the right tool at the right time, and mapfile seems not to be the right tool for my problem, but I will just give you some facts of my observations: using a fast tool like egrep just to find a simple string in my

using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-26 Thread Lennart Schultz
Configuration Information [Automatically generated, do not change]: Machine: i686 OS: linux-gnu Compiler: i686-pc-linux-gnu-gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' -DCONF_OSTYPE='linu x-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-26 Thread Greg Wooledge
On Thu, Mar 26, 2009 at 08:53:50AM +0100, Lennart Schultz wrote: I have a bash script which reads about 25 lines of xml code generating ... mapfile file for i in ${mapfi...@]} do line=$(echo $i) # strip leading blanks case $line in done With this change the job now

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-26 Thread Chet Ramey
Lennart Schultz wrote: Bash Version: 4.0 Patch Level: 10 Release Status: release Description: I have a bash script which reads about 25 lines of xml code generating about 850 files with information extracted from the xml file. It uses the construct: while read line do case

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-26 Thread Chris F.A. Johnson
On Thu, 26 Mar 2009, Lennart Schultz wrote: I have a bash script which reads about 25 lines of xml code generating about 850 files with information extracted from the xml file. It uses the construct: while read line do case $line in done file and this takes a little less than 2

Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

2009-03-26 Thread Chet Ramey
Chris F.A. Johnson wrote: Chet, how about an option to mapfile that strips leading and/or trailing spaces? Another useful option would be to remove newlines. I'm disinclined to add one, since it's easy enough to use the ${line##[ ]} and ${line%%[]} constructs to