Re: piping stderr to tee log (so I can have my log and watch it, too)

[dan mclaughlin]

On Mon, 18 Jan 2016 10:09:14 +0900 Joel Rees  wrote:
> Trying to put some scripts together so I can set an update going one
> night, check it in the morning, reboot, and finish the update while
> I'm at work.
> 
> So I want to do something like
> 
>cd /usr/src && cvs -d$CVSROOT up -Pd | tee /var/log/build/cvssrc.log
>cd /usr/xenocara && cvs -d$CVSROOT up -Pd | tee
> /var/log/build/cvsxenocara.log
>cd /usr/ports && cvs -d$CVSROOT up -Pd | tee /var/log/build/cvsports.log
>cd /usr/src/sys/arch/`machine`/conf && config GENERIC.MP && \
>cd ../compile/GENERIC.MP && make clean && make && \
>make install | tee /var/log/build/buildsys.log 2>&1
>...

i've been down this road, and there are a few problems with this. you don't
check to see if the commands fail before you move on, so if cvs fails, it
will continue with the next cvs or make, etc.

when you pipe something, the error code will come from the last command in
the pipe, which would be tee, so you can't test the error code directly.
one way around this is what i do for commands that fail is i have some
functions:

make_failed() {
  [ $# -ge 1 ] || return 1
  [ ${#1} -gt 0 ] || return 1
  tail -1 "$1" | grep -q "\*\*\* Error "
}
cvs_failed()
  [ $# -ge 1 ] || return 1
  [ ${#1} -gt 0 ] || return 1
  tail -1 "$1" | grep -q "\[update aborted\]"
}

which will return true if the last line has an error (since a function
returns the error code of the last program executed, and grep here fails
or not depending on the presence of the error string.)

i use it like this:

make 2>&1 | tee $_logfile_build
make_failed $_logfile_build && errx "make failed"

or using your example:

cd /usr/src && cvs -d$CVSROOT up -Pd 2>&1 | tee /var/log/build/cvssrc.log
cvs_failed /var/log/build/cvsrsc.log && errx "cvs src failed"

where errx is another function of mine:

err() { echo "$0: ERR $*" >&2; }
errx() { echo "$0: ERR $*" >&2; exit 1; }

i have an include file with basic functions i use in many of my scripts like
this, at the head of the script i have a line:

. ~/.func


a few more notes on the shell. you can do something like

if ! make;then
  err "make failed"
fi 2>&1 | tee logfile

which will send all output from the 'if' statement to tee, much like the
{ cmd1; cmd2; } examples you were given earlier. one issue with this is
that when you pipe it like this, it spawns a subshell, so nothing in the
if statement goes outside. an 'exit' will only exit the 'if' statement (or
'while', 'for', etc) and not the program, so my 'errx' function above does
not exit the whole script, only the subshell.

the same holds true for '(cmd)' statements, which start a subshell, which
is why you can do (cd /usr/src && ls) and it returns to its original directory
afterwards, since the 'cd' only changes the subshell's working directory.

you also can't set varibles in '(cmd)' or in anything in a pipeline (like
that 'if'), and have them carry over to the rest of the script. you CAN
however set variables in '{ cmd; }' statements.

a good place to learn some interesting shell techiniques in in things like
the /etc/rc* scripts and the install scripts in /usr/src/distrib/miniroot/.
sometimes you learn the hard way though through experimentation and failure
(even if you RTFM sometimes the meaning only becomes clear after you
experience it.)

Re: piping stderr to tee log (so I can have my log and watch it, too)

[Claus Assmann]

On Mon, Jan 18, 2016, Joel Rees wrote:

>cd /usr/src && cvs -d$CVSROOT up -Pd | tee /var/log/build/cvssrc.log

> except the 2>&1 is, I think the book says, too late to collect both

Which book?

> output streams into buildsys.log .

cd /usr/src && cvs -d$CVSROOT up -Pd 2>&1 | tee /var/log/build/cvssrc.log

Re: piping stderr to tee log (so I can have my log and watch it, too)

[Joel Rees]

Well, after posting this

On Mon, Jan 18, 2016 at 10:09 AM, Joel Rees  wrote:
> Trying to put some scripts together so I can set an update going one
> night, check it in the morning, reboot, and finish the update while
> I'm at work.
>
> So I want to do something like
>
>cd /usr/src && cvs -d$CVSROOT up -Pd | tee /var/log/build/cvssrc.log
>cd /usr/xenocara && cvs -d$CVSROOT up -Pd | tee
> /var/log/build/cvsxenocara.log
>cd /usr/ports && cvs -d$CVSROOT up -Pd | tee /var/log/build/cvsports.log
>cd /usr/src/sys/arch/`machine`/conf && config GENERIC.MP && \
>cd ../compile/GENERIC.MP && make clean && make && \
>make install | tee /var/log/build/buildsys.log 2>&1
>...
>
> except the 2>&1 is, I think the book says, too late to collect both
> output streams into buildsys.log .
>
> I found
>
>exec > >(tee ${LOGFILE}) 2>&1
>
> suggested on stackexchange (with claims that it works in ksh), but a
> simple test with
>
>ls /nonexisting > >(tee mylog) 2>&1
>
> fails with
>
>ksh: syntax error: `> ' unexpected
>
> Any suggestions appreciated. Cluebats, too.

I checked over the answer on stackexchange, played around with an
example of juggling file descriptors, and found this seems to work:

   { ls ${DIR} ; } 2>&1 | tee mylog

catching the output whether DIR exists or not. So I'm going to try

   { rm -rf /usr/obj/* && cd /usr/src && make obj && \
   cd /usr/sr/etc && env DESTDIR=/ make distrib-dirs && \
   cd /usr/src && make build ; } 2>&1 | tee /var/log/build/buildsys.log

in a script by itself now.

Sorry for talking to myself on list

-- 
Joel Rees

Be careful when you look at conspiracy.
Arm yourself with knowledge of yourself, as well:
http://reiisi.blogspot.jp/2011/10/conspiracy-theories.html

Re: piping stderr to tee log (so I can have my log and watch it, too)

[Delan Azabani]

Put the commands that are to be logged in a grouping command, and
then apply 2>&1 to that grouping command:

{ cd ../compile/GENERIC.MP && \
  make clean && make && make install; } \
  2>&1 | tee /var/log/build/buildsys.log

Note that grouping commands that use braces (i.e. those that execute
their commands in the current environment) require a semicolon before
they are closed, but those with parentheses do not.

If you want stdout and stderr to be logged to separate files, or
you otherwise want to keep stdout and stderr separate, then a more
complex dance is needed:

{ { cd ../compile/GENERIC.MP && \
  make clean && make && make install; } \
  2>&3 | tee /var/log/build/buildsys.out.log; } \
  3>&1 | tee /var/log/build/buildsys.err.log 1>&2

Re: piping stderr to tee log (so I can have my log and watch it, too)

[Joel Rees]

On Mon, Jan 18, 2016 at 10:38 AM, Delan Azabani  wrote:
> Put the commands that are to be logged in a grouping command, and
> then apply 2>&1 to that grouping command:
>
> { cd ../compile/GENERIC.MP && \
>   make clean && make && make install; } \
>   2>&1 | tee /var/log/build/buildsys.log
>
> Note that grouping commands that use braces (i.e. those that execute
> their commands in the current environment) require a semicolon before
> they are closed, but those with parentheses do not.

This may help me remember grouping syntax for real.

> If you want stdout and stderr to be logged to separate files, or
> you otherwise want to keep stdout and stderr separate, then a more
> complex dance is needed:
>
> { { cd ../compile/GENERIC.MP && \
>   make clean && make && make install; } \
>   2>&3 | tee /var/log/build/buildsys.out.log; } \
>   3>&1 | tee /var/log/build/buildsys.err.log 1>&2

Thanks, Delan (and Nigel, off list, for the parentheses example).

-- 
Joel Rees