Re: need help making shell script use two CPUs/cores

2011-01-25 Thread Stan Hoeppner
Carl Johnson put forth on 1/24/2011 5:07 PM: Stan Hoeppner s...@hardwarefreak.com writes: Now we have 4 CPUs on two memory channels. If not for caches, you'd see no speedup past 2 Imagemagick processes. Which is pretty much the behavior identified by another OP with an Athlon II

Re: need help making shell script use two CPUs/cores

2011-01-24 Thread Bob Proulx
Stan Hoeppner wrote: Bob Proulx put forth: Here is some raw data from another test using GraphicsMagic from Debian Sid on an Intel Core2 Quad CPU Q9400 @ 2.66GHz. #CPUs real user sys 1 ... 32.17 100.15 2.29 2 ... 28.02 102.09 2.25 3 ... 26.96 101.41 2.02 4

Re: need help making shell script use two CPUs/cores

2011-01-24 Thread Stan Hoeppner
Bob Proulx put forth on 1/24/2011 12:21 PM: Stan Hoeppner wrote: Why bother going up to 12 processes with a quad core chip? Anything over 4 processes/threads won't gain you anything, as your results above demonstrate. I went to 12 because it would demonstrate the behavior three times

Re: need help making shell script use two CPUs/cores

2011-01-24 Thread Carl Johnson
Stan Hoeppner s...@hardwarefreak.com writes: Now we have 4 CPUs on two memory channels. If not for caches, you'd see no speedup past 2 Imagemagick processes. Which is pretty much the behavior identified by another OP with an Athlon II x4 system--almost zeo speedup from 2 to 4

Re: need help making shell script use two CPUs/cores

2011-01-23 Thread Bob Proulx
Carl Johnson wrote: #CPUs time theoretical time-theoreticalgain/CPU(theoretical) 1 66 2 3666/2 = 33 36-33 = 3 (+9%) 1 -1/2 = 1/2 3 2566/3 = 22 25-22 = 3 (+14%)1/2-1/3 = 1/6 4 2066/4 = 16.5 20-16.5 = 3.5 (+21%)1/3-1/4 =

Re: need help making shell script use two CPUs/cores

2011-01-23 Thread Stan Hoeppner
Bob Proulx put forth on 1/23/2011 8:16 PM: Apparently I've missed some of the thread since my earlier participation. Carl Johnson wrote: #CPUs time theoretical time-theoreticalgain/CPU(theoretical) 1 66 2 3666/2 = 33 36-33 = 3 (+9%) 1 -1/2 = 1/2 3

Re: need help making shell script use two CPUs/cores

2011-01-15 Thread Carl Johnson
Stan Hoeppner s...@hardwarefreak.com writes: Carl Johnson put forth on 1/13/2011 11:34 AM: Processors Time (seconds) P1 66 P2 36 P3 25 P4 20 P5 20 P6 20 P7 20 P8 20 Your

Re: need help making shell script use two CPUs/cores

2011-01-14 Thread Osamu Aoki
On Sun, Jan 09, 2011 at 10:05:43AM -0600, Stan Hoeppner wrote: I'm not very skilled at writing shell scripts. #! /bin/sh for k in $(ls *.JPG); do convert $k -resize 1024 $k; done I use the above script to batch re-size digital camera photos after I dump them to my web server. It takes a

Re: need help making shell script use two CPUs/cores

2011-01-14 Thread Stan Hoeppner
Carl Johnson put forth on 1/13/2011 11:34 AM: Processors Time (seconds) P1 66 P2 36 P3 25 P4 20 P5 20 P6 20 P7 20 P8 20 I am sure the time would have increased if the system

Re: need help making shell script use two CPUs/cores

2011-01-13 Thread Stan Hoeppner
Bob Proulx put forth on 1/12/2011 2:48 PM: That makes a lot of sense to me. And also when cpu time divides by 1/N where N is the number of processes then if you have more convert processes running then effectively that task will get more total time than will the other tasks. A little bit

Re: need help making shell script use two CPUs/cores

2011-01-13 Thread Carl Johnson
Stan Hoeppner s...@hardwarefreak.com writes: Depending on the size of the photos one is converting, if they're relatively small like my 8.3MP 1.8MB jpegs, I'd think something like a dual core Phenom II X2 w/ 6MB L3 cache and 21.4 GB/s memory b/w would likely continue to scale with

Re: need help making shell script use two CPUs/cores

2011-01-12 Thread Camaleón
On Tue, 11 Jan 2011 15:58:45 -0600, Stan Hoeppner wrote: Camaleón put forth on 1/11/2011 9:38 AM: I supposed you wouldn't care much in getting a script to run faster with all the available core occupied if you had a modern (4 years) cpu and plenty of speedy ram because the routine you

Re: need help making shell script use two CPUs/cores

2011-01-12 Thread Bob Proulx
Stan Hoeppner wrote: Bob Proulx put forth: when otherwise it would be waiting for the disk. I believe what you are seeing above is the result of being able to compute during that small block on I/O wait for the disk interval. That's gotta be a very small iowait interval. So small, in

Re: need help making shell script use two CPUs/cores

2011-01-12 Thread Stan Hoeppner
Camaleón put forth on 1/12/2011 3:56 AM: On Tue, 11 Jan 2011 15:58:45 -0600, Stan Hoeppner wrote: Camaleón put forth on 1/11/2011 9:38 AM: I supposed you wouldn't care much in getting a script to run faster with all the available core occupied if you had a modern (4 years) cpu and plenty

Re: need help making shell script use two CPUs/cores

2011-01-12 Thread Stan Hoeppner
Bob Proulx put forth on 1/12/2011 1:11 PM: Stan Hoeppner wrote: Bob Proulx put forth: when otherwise it would be waiting for the disk. I believe what you are seeing above is the result of being able to compute during that small block on I/O wait for the disk interval. That's gotta be a

Re: need help making shell script use two CPUs/cores

2011-01-12 Thread Bob Proulx
Stan Hoeppner wrote: Frankly I'd rather not waste the time on it at this point. You solved my original problem Bob! Thank again. That was the important takeaway here. Now we're into minutia (which can be fun but I'm spending way too much time on debian-user email the last few days) Glad

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Stan Hoeppner
Camaleón put forth on 1/10/2011 2:11 PM: Did'nt you run any test? Okay... (now downloading the sample images) Yes, or course. I just didn't capture results to file. And it's usually better if people see their own results instead of someone else' copy/paste. 2. On your dual processor, or

Re: [OT]: Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Stan Hoeppner
Dan Serban put forth on 1/10/2011 7:52 PM: On Mon, 10 Jan 2011 12:04:19 -0600 Stan Hoeppner s...@hardwarefreak.com wrote: [snip] http://www.hardwarefreak.com/server-pics/ Which gallery system are you using? I quite like it. That's the result of Curator: http://furius.ca/curator/ I've

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Camaleón
On Tue, 11 Jan 2011 07:13:47 -0600, Stan Hoeppner wrote: Camaleón put forth on 1/10/2011 2:11 PM: I used a VM to get the closest environment as you seem to have (a low resource machine) and the above command (timed) gives: I'm not sure what you mean by resources in this context. My box

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Bob Proulx
Stan Hoeppner wrote: Camaleón put forth: real1m44.038s user2m5.420s sys 1m17.561s It uses 2 convert proccesses so the files are being run on pairs. And you can even get the job done faster if using -P8: real1m25.255s user2m1.792s sys

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Bob Proulx
Camaleón wrote: No... I guess this is quite similar to the way most of the daemons do when running in background and launch several instances (like amavisd- new does) That is an optimization to help with the latency overhead associated with forking processes. In order to reduce the response

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Stan Hoeppner
Camaleón put forth on 1/11/2011 9:38 AM: I supposed you wouldn't care much in getting a script to run faster with all the available core occupied if you had a modern (4 years) cpu and plenty of speedy ram because the routine you wanted to run it should not take many time... unless you were

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread John Hasler
Bob writes: They do consume memory and cpu scheduling queue resources. Very little, due to shared memory and copy-on-write. -- John Hasler -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive:

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread John Hasler
Bob writes: Another negative is that other tasks then suffer. That's what group scheduling is for. -- John Hasler -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive:

Re: [OT]: Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Dan Serban
On Tue, 11 Jan 2011 09:18:48 -0600 Stan Hoeppner s...@hardwarefreak.com wrote: Dan Serban put forth on 1/10/2011 7:52 PM: On Mon, 10 Jan 2011 12:04:19 -0600 Stan Hoeppner s...@hardwarefreak.com wrote: [snip] http://www.hardwarefreak.com/server-pics/ Which gallery system are you

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Stan Hoeppner
Bob Proulx put forth on 1/11/2011 3:08 PM: Stan Hoeppner wrote: Camaleón put forth: real1m44.038s user2m5.420s sys 1m17.561s It uses 2 convert proccesses so the files are being run on pairs. And you can even get the job done faster if using -P8: real1m25.255s

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread Stan Hoeppner
John Hasler put forth on 1/11/2011 4:12 PM: Bob writes: They do consume memory and cpu scheduling queue resources. Very little, due to shared memory and copy-on-write. In this case I don't think all that much memory is shared. Each process' data portion is different as each processes a

Re: need help making shell script use two CPUs/cores

2011-01-11 Thread John Hasler
Bob writes: They do consume memory and cpu scheduling queue resources. I wrote: Very little, due to shared memory and copy-on-write. Stan writes: In this case I don't think all that much memory is shared. Each process' data portion is different as each processes a different picture file.

Re: need help making shell script use two CPUs/cores

2011-01-10 Thread Stan Hoeppner
Karl Vogel put forth on 1/9/2011 6:04 PM: On Sun, 09 Jan 2011 10:05:43 -0600, Stan Hoeppner s...@hardwarefreak.com said: S #! /bin/sh S for k in $(ls *.JPG); do convert $k -resize 1024 $k; done Someone was ragging on you to let the shell do the file expansion. I like your way

Re: need help making shell script use two CPUs/cores

2011-01-10 Thread Camaleón
On Sun, 09 Jan 2011 14:39:56 -0600, Stan Hoeppner wrote: Camaleón put forth on 1/9/2011 12:12 PM: Better if you check it, but I dunno how to get the compile options for the lenny package... where is this defined, in source or diff packages? You're taking this thread down the wrong path.

Re: need help making shell script use two CPUs/cores

2011-01-10 Thread Stan Hoeppner
Camaleón put forth on 1/10/2011 8:08 AM: On Sun, 09 Jan 2011 14:39:56 -0600, Stan Hoeppner wrote: Camaleón put forth on 1/9/2011 12:12 PM: Better if you check it, but I dunno how to get the compile options for the lenny package... where is this defined, in source or diff packages? You're

Re: need help making shell script use two CPUs/cores

2011-01-10 Thread Camaleón
On Mon, 10 Jan 2011 12:04:19 -0600, Stan Hoeppner wrote: Camaleón put forth on 1/10/2011 8:08 AM: Good. It would be nice to see the results when you finally go it working the way you like ;-) Bob's xargs suggestion got it working instantly many hours ago. I'm not sure of the results you

[OT]: Re: need help making shell script use two CPUs/cores

2011-01-10 Thread Dan Serban
On Mon, 10 Jan 2011 12:04:19 -0600 Stan Hoeppner s...@hardwarefreak.com wrote: [snip] http://www.hardwarefreak.com/server-pics/ Which gallery system are you using? I quite like it. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble?

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread David Sastre
On Sun, Jan 09, 2011 at 10:05:43AM -0600, Stan Hoeppner wrote: #! /bin/sh for k in $(ls *.JPG); do convert $k -resize 1024 $k; done I use the above script to batch re-size digital camera photos after I dump them to my web server. It takes a very long time with lots of new photos as the

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Camaleón
On Sun, 09 Jan 2011 10:05:43 -0600, Stan Hoeppner wrote: I'm not very skilled at writing shell scripts. #! /bin/sh for k in $(ls *.JPG); do convert $k -resize 1024 $k; done I use the above script to batch re-size digital camera photos after I dump them to my web server. It takes a very

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Stan Hoeppner
Camaleón put forth on 1/9/2011 10:59 AM: http://www.imagemagick.org/Usage/api/#speed The above doc provides hints on how to speed-up image magick operations. Note that multi-threading should be automatically used whether possible, as per this paragraph: *** # IM by default uses

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Camaleón
On Sun, 09 Jan 2011 11:17:04 -0600, Stan Hoeppner wrote: Camaleón put forth on 1/9/2011 10:59 AM: *** # IM by default uses multiple threads for image processing operations. That means you can have the computer do two or more separate threads of image processing, it will be faster than a

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Stan Hoeppner
Camaleón put forth on 1/9/2011 12:12 PM: Better if you check it, but I dunno how to get the compile options for the lenny package... where is this defined, in source or diff packages? You're taking this thread down the wrong path. I asked for assistance writing a simple script to do what I

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Bob Proulx
Stan Hoeppner wrote: I'm not very skilled at writing shell scripts. #! /bin/sh for k in $(ls *.JPG); do convert $k -resize 1024 $k; done First off don't use ls to list files matching a pattern. Instead let the shell match the pattern. #! /bin/sh for k in *.JPG; do convert $k -resize

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Karl Vogel
On Sun, 09 Jan 2011 10:05:43 -0600, Stan Hoeppner s...@hardwarefreak.com said: S #! /bin/sh S for k in $(ls *.JPG); do convert $k -resize 1024 $k; done Someone was ragging on you to let the shell do the file expansion. I like your way better because most scripting shells aren't smart

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Bob Proulx
Karl Vogel wrote: Stan Hoeppner said: S for k in $(ls *.JPG); do convert $k -resize 1024 $k; done Someone was ragging on you to let the shell do the file expansion. I like your way better because most scripting shells aren't smart enough to realize that when there aren't any .JPG

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread shawn wilson
On Jan 9, 2011 3:09 PM, Stan Hoeppner s...@hardwarefreak.com wrote: shawn wilson put forth on 1/9/2011 11:43 AM: On Jan 9, 2011 12:17 PM, Stan Hoeppner s...@hardwarefreak.com wrote: Camaleón put forth on 1/9/2011 10:59 AM: http://www.imagemagick.org/Usage/api/#speed The above doc

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Stan Hoeppner
Bob Proulx put forth on 1/9/2011 3:12 PM: GNU xargs has an extension to run jobs in parallel. This is already installed on your system. (But won't work on other Unix systems.) for k in *.JPG; do echo $k; done | xargs -I{} -P4 echo convert {} -resize 1024 {} Verify that does what you

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Stefan Monnier
unfortunately that simple approach is harder to do with my renaming scheme. So I would probably write a helper script that did the options to convert and renamed the file and so forth. for k in *.JPG; do base=$(basename $k .JPG) test -f $base.1024.jpg continue # skip if already

Re: need help making shell script use two CPUs/cores

2011-01-09 Thread Boyd Stephen Smith Jr.
In 2011011500.46f09b...@kev.msw.wpafb.af.mil, Karl Vogel wrote: On Sun, 09 Jan 2011 10:05:43 -0600, Stan Hoeppner s...@hardwarefreak.com said: S #! /bin/sh S for k in $(ls *.JPG); do convert $k -resize 1024 $k; done Someone was ragging on you to let the shell do the file expansion. I