Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-22 Thread Bill Ricker
On Tue, Oct 21, 2008 at 6:51 PM, Ben Scott [EMAIL PROTECTED] wrote: Perl is poor at SMP (gah! perl threads!). I've never had to worry about Perl MP. Sounds like I should be glad. :-) MP in any language is tricky, but sometimes it appears easy and bites you later. Perl has tried a couple

Re: shell, perl, performance, parallelism, profiling, etc.

2008-10-22 Thread Jerry Feldman
On 10/21/2008 04:02 PM, Ben Scott wrote: According to everything I've ever read, Linux ignores the sticky bit on executables. Wikipedia has a good summary: http://en.wikipedia.org/wiki/Sticky_bit The original reason for the sticky bit is because some Unix commands are so frequently used

Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-22 Thread Ben Scott
On Wed, Oct 22, 2008 at 9:39 AM, Bill Ricker [EMAIL PROTECTED] wrote: MP in any language is tricky ... I've done very little MP work, but that has been my impression. Fundamental to the thinking of most programmers (myself included) is that variables won't change unless the code writes to

Re: shell, perl, performance, parallelism, profiling, etc.

2008-10-22 Thread Ben Scott
On Wed, Oct 22, 2008 at 10:12 AM, Jerry Feldman [EMAIL PROTECTED] wrote: The original reason for the sticky bit is because some Unix commands are so frequently used and small that keeping them in memory significantly improves performance, especially in a multi-user system. Plus (from what

RE: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-22 Thread Michael Pelletier
, performance, parallelism, profiling,etc. (was: Upgrade guidance) On Wed, Oct 22, 2008 at 9:39 AM, Bill Ricker [EMAIL PROTECTED] wrote: MP in any language is tricky ... I've done very little MP work, but that has been my impression. Fundamental to the thinking of most programmers (myself included

Re: shell, perl, performance, parallelism, profiling, etc.

2008-10-22 Thread Jerry Feldman
On 10/22/2008 11:34 AM, Ben Scott wrote: On Wed, Oct 22, 2008 at 10:12 AM, Jerry Feldman [EMAIL PROTECTED] wrote: The original reason for the sticky bit is because some Unix commands are so frequently used and small that keeping them in memory significantly improves performance, especially

Re: shell, perl, performance, parallelism, profiling, etc.

2008-10-22 Thread Jerry Feldman
On 10/22/2008 11:34 AM, Ben Scott wrote: I ran did several test runs in succession. The goal is to get everything cached in RAM for these trials. (Artificial, to be sure, but it's harder to normalize the effects of disk I/O.) The first run in a series was typically different. I threw those

Re: shell, perl, performance, parallelism, profiling, etc.

2008-10-22 Thread Kevin D. Clark
Jerry Feldman writes: However, my thought is that if you are concerned with the performance of a shell or Perl script, then you have the wrong language, and should consider a more traditional compiled language. On the BLU server, we had a script that would convert mailman passwords to

Re: shell, perl, performance, parallelism, profiling, etc.

2008-10-22 Thread Jerry Feldman
On 10/22/2008 03:49 PM, Kevin D. Clark wrote: Jerry Feldman writes: However, my thought is that if you are concerned with the performance of a shell or Perl script, then you have the wrong language, and should consider a more traditional compiled language. On the BLU server, we had a script

RE: shell, perl, performance, parallelism, profiling, etc.

2008-10-22 Thread Roger M. Levasseur
One might remember the Unix vfork(2) system call. Originally, the fork(2) call would clone the entire data segment (data abd .bss), but vfork(2) would implement a copy-on-write strategy. Since Linux has been virtual since inception, there is no difference between the vfork(2) and

shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-21 Thread Ben Scott
On Tue, Oct 21, 2008 at 12:05 PM, Bill McGonigle [EMAIL PROTECTED] wrote: I think we need keywords/tags for 'man -k' to use. Well, there's man -K, but it's slw. I typically use Google as a substitute, but that failed me in this case. I suspect I'd still fail though, even with a better

Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-21 Thread Michael Pelletier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 21 Oct 2008 13:56:21 -0400 Ben Scott [EMAIL PROTECTED] wrote: On Tue, Oct 21, 2008 at 12:05 PM, Bill McGonigle [EMAIL PROTECTED] wrote: Efficiency is surprisingly worse with bash/comm, I don't get why: *Very* interesting. I don't get

Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-21 Thread Michael ODonnell
Note, only valid in bash, not sh. Yah, and it doesn't matter if you use #!/bin/bash at the top if you're running the script with sh foo.sh. It took me five minutes to figure that out just now. D'oh. That process substitution facility is a favorite of mine but it isn't POSIX, and IIRC

Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-21 Thread Bill McGonigle
On Oct 21, 2008, at 13:56, Ben Scott wrote: On Tue, Oct 21, 2008 at 12:05 PM, Bill McGonigle [EMAIL PROTECTED] wrote: I think we need keywords/tags for 'man -k' to use. Well, there's man -K, but it's slw. I typically use Google as a substitute, but that failed me in this case. I

Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-21 Thread Kevin D. Clark
Ben Scott writes: Consider: If the RPM database is in cache, then the package-cleanup utility is going to mainly be reading data from already cached pages (memory mapped file I/O). In the Perl script, everything is serialized. But in the bash script, both package-cleanup invocations can

Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-21 Thread Ben Scott
On Tue, Oct 21, 2008 at 4:24 PM, Bill McGonigle [EMAIL PROTECTED] wrote: Is it because the shell tools fork, and children aren't counted? If that were so my shell numbers should be lower. Good point. As you suggest, maybe time(1) is just borken on MP. Neither the time(1) man page, nor the

Re: shell, perl, performance, parallelism, profiling, etc. (was: Upgrade guidance)

2008-10-21 Thread Ben Scott
On Tue, Oct 21, 2008 at 6:51 PM, Ben Scott [EMAIL PROTECTED] wrote: Challenge accepted okay, intersect.pl will follow in a separate message. #!/usr/bin/perl -w # intersect.pl # Gives the intersection of two text files (lines contained in both files). # Similar to comm(1), but does not