John Gillings asked me about an interesting behavior with a major
performance difference using pipes.
-- part 1 --
Trying this I ran into something silly.
I'm sure I'm having a mental block.
What am I doing wrong?
The 'stuff' I am doing below is just busy work to mimic real work.
It just counts records and matches and tries to get a coarse timing
indication using time().
I'm having trouble using time() within the for loop.
The reason I want it within is for usage during: $ perl -ne "do-stuff-
for-each }{ report-result-at-end "
good (12 seconds):
$ perl -e "$x=-time; for (qx(type MYSCRATCH:tmp.tmp)) {$i++; $n++ if /
hein/i; }$x+=time; print qq($i $n $x)"
42979 10 12
bad (0 seconds):
$ perl -e "for (qx(type MYSCRATCH:tmp.tmp)) {$x=-time unless $i++; $n+
+ if /hein/i; }$x+=time; print qq($i $n $x)"
42979 10 0
-- part 2 --
Here are a few more details on why I stumbled into this.
These are sloppy tests for now.
Just concerned with orders of magnitude.
Perl V5.10.0, 2cpu DS20, V8.3, uncontrolled, busy background.
1) process 50K records from a file directly: 1 second :
perl -ne "stuff" file.dat
10) process the same 50K records through a subprocess mailbox: 10
seconds ...
This is surely slow due to the 100,000 mailbox QIOs. Acceptable.
perl -e "for (qx(type file.dat){stuff}"
100) process the same 50K records piped to perl: 100 seconds.
This causes RWMBX conditions. Not workable.
$ pipe type file.dat | perl -ne "stuff"
Just thinking out aloud....
Apparently the DCL PIPE sender is using IO$M_NOW, not waiting for the
message to be read.
It seems clear to me that if the receiver is slower than the send that
the mailbox will fill up, no matter how big it is, resulting in the
sender process being put in RWMBX state (SCH$RESOURCE_WAIT) .
I guess I'm wondering when/why/by-who SCH$RAVAIL will be called to put
it back in COM mode.
Apparently the mailbox driver does not do this right when a message is
read and (some) room is made. This fill-fill-fill, bump, freeze, suck-
suck-suck, release would be inefficient, but it should move forward.
As it is, the whole process does move forward but very slowly so, and
not consuming any cpu resources.
Lowering process priority on the writer seems to help a little. To be
verified (if I feel like it)
SYSGEN DEFMBXBUFQUO=1056. I tried 64000 briefly but that did not seem
to help
Also tried large settings for DECC$PIPE_BUFFER_QUOTA / SIZE 'just in
case'
Those test where not robust though, may well have done something
wrong.
Some reading
http://h71000.www7.hp.com/openvms/journal/v9/mailboxes.pdf (Bruce
Ellis)
OpenVMS Alpha Internals V7.0 (Goldenberg/Dumas/Saravanan)
2.4.3.3.1 "System Resource Miscellaneous Waits"
2.6.1.5 "Context for an MWAIT Wait"
Comments?
Hein.
