Re: Shell scripts: variable assignment within read loops
On Sun, 17 Aug 2008, David Wolfskill wrote: [snipped] will assign to foo the value of the bar variable form the last record read (in FreeBSD 6.3-STABLE, at least), the following fails to do so: foo= cat $filename | while read bar ... ; do ... foo=$bar ... done echo $foo Well, that's not *quite* accurate:the assignment is done all right, but in the latter case, it appears to be done in a subshell, so by the time we get to the echo statement, any variable assignments from within the read loop have vanished. You've already defined the reason behind this. Since subshells for parts of pipelines aren't guaranteed, you need something more clever. Assuming you're only interested in the _last_ value of foo, you could try this: foo=$( cat $filename | ( while read bar; do ... foo=$bar ... done echo $foo ) ) Cheers, jan -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ ( echo ouroboros; cat ) /dev/fd/0 # it's like talking to yourself sometimes ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Shell scripts: variable assignment within read loops
David Wolfskill wrote: I am writing a (Bourne) shell script that is intended (among other things) to obtain information from a command, such as: netstat -nibd -f inet by reading and parsing the output. However, the obvious (to me) approach of piping the output of the command to the standard input of a while read ... statement turns out to be not very useful; [...] Well, that's not *quite* accurate:the assignment is done all right, but in the latter case, it appears to be done in a subshell, so by the time we get to the echo statement, any variable assignments from within the read loop have vanished. That's correct, as Giorgos has already pointed out. Most bourne shells execute all parts of a pipe except the first one in a subshell, so any assignments are lost. A common way is to echo things from within the subshell and capture them through command expansion, like this: foo=$( something | while read x; do whatever echo value done ) That will assign value to the variable foo. This only works for single variables, of course. If you want to assign to multiple variables, it gets a little more tricky. One way is to use single assignment like above, and then split it into several variables on a delimiter character. The following will split $foo on whitespace and assign the results to $1, $2, $3 etc., with the count in $#: set -- $foo You can split on any other character by setting the IFS variable of the shell appropriately. If you know in advance how many values you'll get, another possibility is to use a so-called here document: read foo1 foo2 foo3 rest end $( something | while read x; do whatever echo -n value done ) end For example, a simple way to get hour, minutes and seconds into three variables without having to exec date(1) three times: read H M S end $( date +%H %M %S ) end Or: set -- $( date +%H %M %S ) H=$1 M=$2 S=$3 It gets more complicated if you need to get the exit code of some parts of the pipe except the last one. You didn't ask for that, though. :-) Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd 'Instead of asking why a piece of software is using 1970s technology, start asking why software is ignoring 30 years of accumulated wisdom.' ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Shell scripts: variable assignment within read loops
David Wolfskill wrote:
foo=
cat $filename | while read bar ... ; do
...
foo=$bar
...
done
echo $foo
A trick I've used to great advantage in bourne shell and bash for
passing multiple variables back is to produce small snippets of shell
script within a function, such as the following, for pulling in a bunch
of variables with a single program invocation for efficiency:
get_stats(){
stat -fc 'mount=%n blksz=%S total=%b free_root=%f \
free_user=%a' [EMAIL PROTECTED]
}
get_stats / /dev /tmp | while read line
do
eval ${line}
# now mount, blksz, total, free_root, and free_user are set here.
printf = %s has %u free %u-byte blocks, out of %u\n \
${mount} ${free_user} ${blksz} ${total}
done
The function returns a series of lines that can be iterated with 'while
read', and evaluated individually for action. If the function returns
only a single line, it can be passed directly into eval:
eval $(get_stats /)
As this shortcut does execute arbitrary code, however, there is always a
chance that it can be hijacked for nefarious purposes if the data source
is untrusted.
--
Fuzzy love,
-CyberLeo
Technical Administrator
CyberLeo.Net Webhosting
http://www.CyberLeo.Net
[EMAIL PROTECTED]
Furry Peace! - http://.fur.com/peace/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Shell scripts: variable assignment within read loops
As I thought while reading your message, awk seems to be a good solution. Just a note: On Mon, 18 Aug 2008 06:29:03 +0300, Giorgos Keramidas [EMAIL PROTECTED] wrote: Would you be ok with an awk(1) script instead of /bin/sh? It tends to be nicer for this sort of thing, i.e.: [...] $ netstat -nibd -f inet | awk -f david.awk You could start your awk skript with #!/usr/bin/awk and give it +x attribute, as well as adding the desired source command netstat -nibd -f inet to the script, using awk's system() function, so you can start it more easily or use it in combination with other commands. % ./netstuff.awk -- Polytropon From Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Shell scripts: variable assignment within read loops
On Mon, 18 Aug 2008 14:33:05 +0200, Polytropon [EMAIL PROTECTED] wrote: As I thought while reading your message, awk seems to be a good solution. Just a note: On Mon, 18 Aug 2008 06:29:03 +0300, Giorgos Keramidas [EMAIL PROTECTED] wrote: Would you be ok with an awk(1) script instead of /bin/sh? It tends to be nicer for this sort of thing, i.e.: [...] $ netstat -nibd -f inet | awk -f david.awk You could start your awk skript with #!/usr/bin/awk and give it +x attribute, as well as adding the desired source command netstat -nibd -f inet to the script, using awk's system() function, so you can start it more easily or use it in combination with other commands. % ./netstuff.awk That's an option, but piping *to* awk may be slightly trickier then. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Shell scripts: variable assignment within read loops
On Sun, 17 Aug 2008 18:33:28 -0700, David Wolfskill [EMAIL PROTECTED] wrote:
I am writing a (Bourne) shell script that is intended (among other
things) to obtain information from a command, such as:
netstat -nibd -f inet
by reading and parsing the output.
However, the obvious (to me) approach of piping the output of the
command to the standard input of a while read ... statement turns out
to be not very useful; it appears that while
foo=
while read bar ... ; do
...
foo=$bar
...
done $filename
echo $foo
will assign to foo the value of the bar variable form the last record
read (in FreeBSD 6.3-STABLE, at least), the following fails to do so:
foo=
cat $filename | while read bar ... ; do
...
foo=$bar
...
done
echo $foo
Well, that's not *quite* accurate:the assignment is done all right, but
in the latter case, it appears to be done in a subshell, so by the time
we get to the echo statement, any variable assignments from within the
read loop have vanished.
Hi David,
You are right that feeding data to a looping construct through a pipe
may run in a subshell. The ``Single UNIX Specification'' says
... each command of a multi-command pipeline is in a subshell
environment; as an extension, however, any or all commands in a
pipeline may be executed in the current environment.
You can read the online text of the Single UNIX Specification at:
http://www.unix.org/single_unix_specification/
A simple 'registration' is required, and you need a browser that
supports cookies. But after that, you can follow the links to the shell
command language page at
http://www.opengroup.org/onlinepubs/95399/utilities/xcu_chap02.html
Near section `` 2.12 Shell Execution Environment'' you can find the text
I quoted.
What I usually do in similar shell scripts is something like:
cat ${filename} | sed -n -e '/foo/ s/bar/baz/' | \
xargs -n1 blah
This isn't exactly the same as assigning $foo to the results of the
loop, but you can also use:
foo=`cat $filename | while read bar ; do \
stuff ...
echo $bar
more stuff...
done`
Now here's a copy of the in-development script:
#! /bin/sh
cmd=netstat -nibd -f inet
ctr=0
clist=
hlist=`$cmd | head -1`
for f in $hlist; do
ctr=$(( $ctr + 1 ))
eval c$ctr=\$f\
eval h_$f=c$ctr
done
cmax=$ctr
t_file=`mktemp /tmp/X`
$cmd | tail +2 $t_file
while read $hlist dummy; do
if [ $Name = lo0 ]; then
continue
fi
for f in $hlist; do
eval val=\\$$f\
case $val in
-) eval ${f}_$Name=0;;
*) eval ${f}_$Name=$val;;
esac;
done
nics=$Name $nics;
done/$t_file
rm $t_file
echo (end) NICs: $nics
for n in $nics; do
for f in $hlist; do
eval echo ${f}_$n: \$${f}_$n
done
done
exit 0
And its output on my laptop:
(end) NICs: ath0
Name_ath0: ath0
Mtu_ath0: 1500
Network_ath0: 172.17
Address_ath0: 172.17.1.30
Ipkts_ath0: 725191
Ierrs_ath0: 0
Ibytes_ath0: 185144197
Opkts_ath0: 821917
Oerrs_ath0: 0
Obytes_ath0: 74260936
Coll_ath0: 0
Drop_ath0: 0
and (somewhat more interestingly) on my firewall machine:
(end) NICs: dc0 de0 fxp0
Name_dc0: dc0
Mtu_dc0: 1500
Network_dc0: 172.16.8/24
Address_dc0: 172.16.8.1
Ipkts_dc0: 2501577
Ierrs_dc0: 0
Ibytes_dc0: 215386153
Opkts_dc0: 20269087
Oerrs_dc0: 0
Obytes_dc0: 2553930555
Coll_dc0: 0
Drop_dc0: 0
Name_de0: de0
Mtu_de0: 1500
Network_de0: 63.193.123/24
Address_de0: 63.193.123.122
Ipkts_de0: 5936847
Ierrs_de0: 0
Ibytes_de0: 734092787
Opkts_de0: 18557543
Oerrs_de0: 0
Obytes_de0: 2551089632
Coll_de0: 0
Drop_de0: 0
Name_fxp0: fxp0
Mtu_fxp0: 1500
Network_fxp0: 172.17
Address_fxp0: 172.17.0.1
Ipkts_fxp0: 10013
Ierrs_fxp0: 0
Ibytes_fxp0: 1366082
Opkts_fxp0: 1253115
Oerrs_fxp0: 0
Obytes_fxp0: 70429903
Coll_fxp0: 0
Drop_fxp0: 0
As you see, I am circumventing the issue by writing to a transient
file. In the intended application, the script is to be used to gather
resource-utilization information; thus, I want its footprint to be
smaller, rather than larger. Granted, in my case, I would be writing
a tiny text file to a swap-backed tmpfs, but in production, I won't
have the luxury of knowing that in advance: the intent is that the
script must run on a minimal FreeBSD system, with no ports or other
3rd-party software installed.
Is there some other -- possibly better -- way to do this (using Bourne
shell scripting)?
Ah, that's much better. Now I see what you are trying to do. Would you
be ok with an awk(1) script instead of /bin/sh? It tends to be nicer
for this sort of thing, i.e.:
$ expand david.awk | cat -n
1 #
2 # Gather the field names if this is a header-line.
3 #
4 $0 ~ /^Name/ {
5 for (k = 1; k = NF; k++)
6 tag[k] = $k;
7 }
8
9 #
10 #
Re: Shell scripts: variable assignment within read loops
On Mon, Aug 18, 2008 at 06:29:03AM +0300, Giorgos Keramidas wrote:
...
You are right that feeding data to a looping construct through a pipe
may run in a subshell. The ``Single UNIX Specification'' says
Ah; thanks for the confirmation.
...
What I usually do in similar shell scripts is something like:
cat ${filename} | sed -n -e '/foo/ s/bar/baz/' | \
xargs -n1 blah
This isn't exactly the same as assigning $foo to the results of the
loop, but you can also use:
foo=`cat $filename | while read bar ; do \
stuff ...
echo $bar
more stuff...
done`
Right; I had seen that type of construct in /etc/rc.d.* (which is where
I often look for samples of shell scripts that need to work reliably).
As you noticed, that won't quite do for what I'm trying to accomplish here.
...
As you see, I am circumventing the issue by writing to a transient
file. In the intended application, the script is to be used to gather
resource-utilization information; thus, I want its footprint to be
smaller, rather than larger. Granted, in my case, I would be writing
a tiny text file to a swap-backed tmpfs, but in production, I won't
have the luxury of knowing that in advance: the intent is that the
script must run on a minimal FreeBSD system, with no ports or other
3rd-party software installed.
Is there some other -- possibly better -- way to do this (using Bourne
shell scripting)?
Ah, that's much better. Now I see what you are trying to do.
:-)
Would you be ok with an awk(1) script instead of /bin/sh? It tends
to be nicer for this sort of thing, i.e.:
Yes, awk(1) would be OK. I'll be more inclined to use it if I can
figure out a way to use it instead of sed(1) for a very different part
of the script. :-}
$ expand david.awk | cat -n
1 #
2 # Gather the field names if this is a header-line.
3 #
4 $0 ~ /^Name/ {
5 for (k = 1; k = NF; k++)
6 tag[k] = $k;
7 }
8
9 #
10 # For all other lines, just print the tagged field values.
11 #
12 $0 !~ /^Name/ {
13 name = $1;
14 for (k = 1; k = NF; k++) {
15 if ($k == -)
16 $k = 0;
17 printf %s_%s: %s\n, tag[k], name, $k;
18 }
19 }
$ netstat -nibd -f inet | awk -f david.awk
Name_re0: re0
Mtu_re0: 1500
Network_re0: 192.168.1.0/2
...
Very cool; thank you very much! I will study that a bit
(I'd normally do this stuff in Perl, but in addition to the other issues
mentioned earlier, the script will be sleeping most of the time, but
wake up spit out results periodically. The usual case will be every 5
minutes, but I plan to make use of it with radically shorter periods in
certain specialized environments -- such as every 5 seconds. And I
still want it to be low overhead. I also note in passing that in its
production environments, the script's standard output will be
redirected to append to a file on a different machine via an SSH tunnel.)
With a bit of preprocessing, it may be possible to extract the network
names and print the (end) NICs: XXX XXX part too.
Right -- much of the output I demonstrated was strictly for debugging/
expository purposes.
Thanks again, Giorgos!
Peace,
david
--
David H. Wolfskill [EMAIL PROTECTED]
Depriving a girl or boy of an opportunity for education is evil.
See http://www.catwhisker.org/~david/publickey.gpg for my public key.
pgpQZk6NuCeRc.pgp
Description: PGP signature
