Re: [blfs-dev] BASH script help please
On Tue, 2014-08-19 at 00:28 -0500, Bruce Dubbs wrote:
Christopher Gregory wrote:
Hello,
I have finally overcome my 20 year hatred of scripts and am in the
process of creating bash scripts for all the packages in BLFS.
As packages get upgraded all the time, what I need to do is to have wild
cards to match the package tarballs. I have achieved that with regards
to the below example, however some packages have additional characters
such as boost has boost_1_56_0.tar.bz2 so the wild card match of ? would
not work in this case.
I have not been able to work out how to do a match for this type of
file.
Could someone please assist in this? I only want to know how to do it
in bash thanks.
#!/bin/bash
export PROGRAM=apr-1.5.?
tar -xvf $PROGRAM.tar.?z*
pushd $PROGRAM
./configure --prefix=/usr\
--disable-static \
--with-installbuilddir=/usr/share/apr-1/build
make -j2
make install
popd
rm -rf $PROGRAM
Here is what I use:
$ cat make-apr-1.5.0
#!/bin/bash
source /usr/src/stats
###
DIR=`pwd`
PROGRAM=apr-1.5.0
LOG=$DIR/$PROGRAM.log
TITLE=$PROGRAM
TIMEFORMAT=$TIMEFMT $TITLE
BUILDDIR=/tmp/apr
#DEST=$BUILDDIR/install
rm -f $LOG
rm -rf $BUILDDIR
mkdir $BUILDDIR
cd $BUILDDIR
before=`df -k /tmp | grep / | sed -e s/ \{2,\}/ /g | cut -d' ' -f3`
tar -xf $DIR/$PROGRAM.tar.?z* || exit 1
cd $PROGRAM
{ time \
{
echo Making $TITLE
date
./configure --prefix=/usr\
--disable-static \
--with-installbuilddir=/usr/share/apr-1/build
make
#make test
echo BLFS Start INSTALL
$SUDO make DESTDIR=$DEST install
}
} 21 | tee -a $LOG
if [ $PIPESTATUS -ne 0 ]; then exit 1; fi;
stats $LOG $DIR/$PROGRAM.tar.?z* $before
exit 0
$ cat /usr/src/stats
#!/bin/bash
function stats()
{
log=$1
tarball=$2
b4=$3
base_sbu=118
free_now=`df -k /tmp | grep / | sed -e s/ \{2,\}/ /g | cut -d -f3`
buildtime=`tail -n1 $log|cut -f1 -d `
sbu=`echo scale=3; $buildtime / $base_sbu | bc`
psizeK=`du -k $tarball | cut -f1`
psizeM=`echo scale=3; $psizeK / 1024 | bc`
bsizeK=`echo $free_now - $b4 | bc`
bsizeM=`echo scale=3; $bsizeK / 1024 | bc`
echo SBU=$sbu | tee -a $log
echo $psizeK $tarball size ($psizeM MB)| tee -a $log
echo $bsizeK kilobytes build size ($bsizeM MB) | tee -a $log
(echo -n md5sum : ; md5sum $tarball) | tee -a $log
(echo -n sha1sum: ; sha1sum $tarball) | tee -a $log
echo `date` $tarball /usr/src/packages-$(lsb_release -r|cut -f2).log
}
TIMEFMT='%1R Elapsed Time - '
SUDO='sudo -E'
===
base_sbu is the binutils time.
You can modify this to pass parameters, but I generally edit the script
for any new version. The build procedures change often enough that I
generally copy the old build script to the new and update as needed.
I build in /tmp, but that can be modified as desired. I don't delete
until /tmp gets full so I can review if needed.
-- Bruce
Hello Bruce,
Thanks for that. It puts my poor little scripts to shame. Just before
your reply came in, I finally managed to get the wildcards working
correctly by adding another ??. I also found that it helps considerably
when the correct suffix is specified, otherwise the poor script can not
extract the archive, which in turn means that $PROGRAM can never be set
correctly.
I need to get these scripts done, as I am now building often enough that
I am getting sick of copying and pasting. Once they are all done then I
will duplicate them and have one set for doing a permanent install and
the other for doing the calculations.
I am now trying to save up to get a desktop computer. Looking at your
base_sbu highlights just how slow my laptop is. It probably won't be
more than a quad core that I get as I will also need to get a monitor.
An 8 core would be wonderful however I think at the moment out of my
price bracket. Can always get another later.
Regards,
Christopher.
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Re: [blfs-dev] BASH script help please
Christopher Gregory wrote:
Hello,
I have finally overcome my 20 year hatred of scripts and am in the
process of creating bash scripts for all the packages in BLFS.
As packages get upgraded all the time, what I need to do is to have wild
cards to match the package tarballs. I have achieved that with regards
to the below example, however some packages have additional characters
such as boost has boost_1_56_0.tar.bz2 so the wild card match of ? would
not work in this case.
I have not been able to work out how to do a match for this type of
file.
Could someone please assist in this? I only want to know how to do it
in bash thanks.
#!/bin/bash
export PROGRAM=apr-1.5.?
tar -xvf $PROGRAM.tar.?z*
pushd $PROGRAM
./configure --prefix=/usr\
--disable-static \
--with-installbuilddir=/usr/share/apr-1/build
make -j2
make install
popd
rm -rf $PROGRAM
Here is what I use:
$ cat make-apr-1.5.0
#!/bin/bash
source /usr/src/stats
###
DIR=`pwd`
PROGRAM=apr-1.5.0
LOG=$DIR/$PROGRAM.log
TITLE=$PROGRAM
TIMEFORMAT=$TIMEFMT $TITLE
BUILDDIR=/tmp/apr
#DEST=$BUILDDIR/install
rm -f $LOG
rm -rf $BUILDDIR
mkdir $BUILDDIR
cd $BUILDDIR
before=`df -k /tmp | grep / | sed -e s/ \{2,\}/ /g | cut -d' ' -f3`
tar -xf $DIR/$PROGRAM.tar.?z* || exit 1
cd $PROGRAM
{ time \
{
echo Making $TITLE
date
./configure --prefix=/usr\
--disable-static \
--with-installbuilddir=/usr/share/apr-1/build
make
#make test
echo BLFS Start INSTALL
$SUDO make DESTDIR=$DEST install
}
} 21 | tee -a $LOG
if [ $PIPESTATUS -ne 0 ]; then exit 1; fi;
stats $LOG $DIR/$PROGRAM.tar.?z* $before
exit 0
$ cat /usr/src/stats
#!/bin/bash
function stats()
{
log=$1
tarball=$2
b4=$3
base_sbu=118
free_now=`df -k /tmp | grep / | sed -e s/ \{2,\}/ /g | cut -d -f3`
buildtime=`tail -n1 $log|cut -f1 -d `
sbu=`echo scale=3; $buildtime / $base_sbu | bc`
psizeK=`du -k $tarball | cut -f1`
psizeM=`echo scale=3; $psizeK / 1024 | bc`
bsizeK=`echo $free_now - $b4 | bc`
bsizeM=`echo scale=3; $bsizeK / 1024 | bc`
echo SBU=$sbu | tee -a $log
echo $psizeK $tarball size ($psizeM MB)| tee -a $log
echo $bsizeK kilobytes build size ($bsizeM MB) | tee -a $log
(echo -n md5sum : ; md5sum $tarball) | tee -a $log
(echo -n sha1sum: ; sha1sum $tarball) | tee -a $log
echo `date` $tarball /usr/src/packages-$(lsb_release -r|cut -f2).log
}
TIMEFMT='%1R Elapsed Time - '
SUDO='sudo -E'
===
base_sbu is the binutils time.
You can modify this to pass parameters, but I generally edit the script
for any new version. The build procedures change often enough that I
generally copy the old build script to the new and update as needed.
I build in /tmp, but that can be modified as desired. I don't delete
until /tmp gets full so I can review if needed.
-- Bruce
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Re: [blfs-dev] bash script
On Tue, Jun 03, 2014 at 11:16:32AM +1200, Christopher Gregory wrote: It is really weird that the times do vary between runs of the script. On a non-realtime OS you should always expect _some_ variation. If you look in 'top', even on a system without 'cron' or 'at' there will be a lot of processes which might wake up. Also memory pressure if you are doing other things during the build and disk-head movement (on rotating hard disks). For different toolchain versions there will also be differences (usually, newer means slower to compile, but with 4.9 I have seen shorter compiles in some packages. Also, after watching the frequencies of my four AMD cores during compilation with -j1 on a mostly idle system (I was worried about why the ondemand governor showed excessive variation in my SBUs for recent system builds) I realised that work is being shuffled around the cores. On modern intel, the 'performance' governor has replaced 'ondemand' and works as well as ondemand used to (i.e. it quickly speeds up and quickly slows down when the load has gone). ĸen -- Nanny Ogg usually went to bed early. After all, she was an old lady. Sometimes she went to bed as early as 6 a.m. -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] bash script
On Mon, 2 Jun 2014 23:57:40 +1200
Christopher Gregory m...@pc-networking-services.com wrote:
The script at:
http://lists.linuxfromscratch.org/pipermail/alfs-discuss/2005-December/007441.html
which has a few more features than the one on Bruce's SBU page:
http://www.linuxfromscratch.org/~bdubbs/about.html
is what I am using, with the program name modified accordingly.
There has to be a way to just pass the integer value of Elapsed Time
to a variable without getting the frigging error that it is not an
integer value.
See below for one approach.
I know from the research that time outputs to stderr.
This complicates matters somewhat. To say the least..
# ...
{
# ... yada yada yada ...
} 21 | tee $SDIR/logs/$PROGRAM.log
time_elapsed=$(grep 'Elapsed Time' $SDIR/logs/$PROGRAM.log | \
sed 's/^\([[:digit:]]\+\)\..*$/\1/');
# ...
echo SBU: $(($time_elapsed / $standard_sbu))
Now, I tried for an hour or two to figure out a way that would redirect
various outputs and inputs via anonymous pipes and produce both the
effect of filling $time_elapsed and also print out the output on the
screen. I failed. Which was really dissapointing because I think that a
system like Linux should be able to do that. But no matter, the
infomation we need is inside the log file, so you just make a second
pass over the log file when all is done and extract the result.
Maybe perl or tlc would be able to do this task better. The fact that
the shell is unable to do what it SHOULD be able to do is quite
embarrasing indeed.
--
Svi moji e-mailovi su kriptografski potpisani. Proverite ih.
All of my e-mails are cryptographically signed. Verify them.
--
You don't need an AI for a robot uprising.
Humans will do just fine.
signature.asc
Description: PGP signature
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Re: [blfs-dev] bash script
Aleksandar Kuktin wrote: # ... yada yada yada ... } 21 | tee $SDIR/logs/$PROGRAM.log time_elapsed=$(grep 'Elapsed Time' $SDIR/logs/$PROGRAM.log | \ sed 's/^\([[:digit:]]\+\)\..*$/\1/'); # ... echo SBU: $(($time_elapsed / $standard_sbu)) But no matter, the infomation we need is inside the log file, so you just make a second pass over the log file when all is done and extract the result. Right. In my case, I happen to know that the line I need is the last one in the current log, so I use buildtime=`tail -n1 $log|cut -f1 -d ` Note that the time output honors the TIMEFORMAT environment variable, so I am using TIMEFORMAT=%1R Elapsed Time - $TITLE, so I get for output something like '123.4 Elapsed Time - some title' I then use bc to do the math and that gets a bit more accuracy, which helps especially for the packages that are less than 1 SBU. sbu=`echo scale=3; $buildtime / $base_sbu | bc` -- Bruce -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] bash script
On Tue, June 3, 2014 2:23 am, Ken Moffat wrote:
On Mon, Jun 02, 2014 at 11:57:40PM +1200, Christopher Gregory wrote:
Hello,
Well I have spent hours looking through the man pages on time(2), bash,
and trying to modify the SBU calculation script that jhalf uses to
work as a standalone to no avail.
The script at:
http://lists.linuxfromscratch.org/pipermail/alfs-discuss/2005-December/
007441.html
which has a few more features than the one on Bruce's SBU page:
http://www.linuxfromscratch.org/~bdubbs/about.html
is what I am using, with the program name modified accordingly.
what I need is the value of Elapsed Time set to a variable so that I
can divide it by the base_sbu value of bin-utils.
I know that I can manually do this, but I really do need it to work in
a script.
The examples I have found on google are no help to me.
There has to be a way to just pass the integer value of Elapsed Time to
a variable without getting the frigging error that it is not an integer
value.
I know from the research that time outputs to stderr.
I am NOT a scripter nor a very good programmer as I normally hate going
around in endless circles going no where.
Regards,
Christopher.
My current scripts are a bit more involved than that. I still
calculate an SBU value, but only because I used to. In the beginning, I
stored $SECONDS.
Now, I use the following (actually, I have a lot of debug code as
well, and I've reformatted this slightly) -
km_start_SBU () { # start the package's SBU timing from here, after it has
# been untarred
KM_DOING=$(date +%s.%N | awk '{ printf %8.3f, $1 }')
}
km_end_SBU () { # stop the timing : called from km_finish()
KM_DONE=$(date +%s.%N | awk '{ printf %8.3f, $1 }')
}
km_calcSBU() {
# calculate its SBU
# first calculate the elapsed time
KM_MYTIME=$(echo $KM_DOING $KM_DONE | awk '{ printf %8.3f, $2-$1 }')
# if the SBU stamp exists, read it
if [ -f $KM_SBU ]; then KM_SBUTIME=`tail -n 1 $KM_SBU | awk '{ print $NF}'`
# now set the sbu without a newline
KM_MYSBU=`echo $KM_MYTIME $KM_SBUTIME | \
awk '{ sbu = $1 / $2 ; printf(%3.1f,sbu) }'` # minimum value is less than
0.1, not 0.0
if [ $KM_MYSBU = 0.0 ]; then KM_MYSBU=less than 0.1
fi else # set the SBU from first package (assumes a straight
# LFS build)
# and therefore own time is 1.0 SBU
echo $KM_MYTIME $KM_SBU KM_MYSBU=1.0
fi }
The key thing, as with any script, is to use the tools which are
available. In my case, I use awk for doing the maths.
I suggest that whole seconds are probably perfectly fine for most
usage of SBUs, there is always some variation between different runs (and
nowadays the variation has increased if I use the ondemand cpu governor,
so I've now stopped using that for measured builds).
A couple of other notes:
KM_MYSBU points to a file which holds the elapsed time in seconds
and decimals for 1.0 SBU, and the last line is what I use - after the
system is completed, I recalculate the SBU to see how it compares to the
SBU from the host system, so that recalculation uses
its own code to append a new line. That value is what I actually use as
the divisor when measuring packages for the book(s), but for the
measurements themselves I normally do a DESTDIR build and time each step
individually, then use a calculator.
All the KM_ and km_ prefixes are because _something_ broke when
bison got upgraded a year or so ago (i.e. bison would either not compile,
or its testsuite broke, I forget which). One of my variables or functions
had a name which the bison devs had decided to use.
ĸen
--
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Hello Ken,
Thanks for this. Bruce and others have also provided input as well. I
will probably try out this script at some point.
I just want to get a script working correctly for adding pages that are
not in the book.
I am working on a complete gnome desktop blfs book at the moment. I
already had discussions in private with Bruce before Wayne posted his
version.
Mine is strictly by the developers recommendation. A rather large task,
seeing that there are, from memory 274 packages for the complete current
gnome.
Regards,
Christopher.
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Re: [blfs-dev] bash script
On Tue, June 3, 2014 4:25 am, Aleksandar Kuktin wrote:
On Mon, 2 Jun 2014 23:57:40 +1200
Christopher Gregory m...@pc-networking-services.com wrote:
The script at:
http://lists.linuxfromscratch.org/pipermail/alfs-discuss/2005-December/
007441.html
which has a few more features than the one on Bruce's SBU page:
http://www.linuxfromscratch.org/~bdubbs/about.html
is what I am using, with the program name modified accordingly.
There has to be a way to just pass the integer value of Elapsed Time
to a variable without getting the frigging error that it is not an
integer value.
See below for one approach.
I know from the research that time outputs to stderr.
This complicates matters somewhat. To say the least..
# ...
{
# ... yada yada yada ...
} 21 | tee $SDIR/logs/$PROGRAM.log
time_elapsed=$(grep 'Elapsed Time' $SDIR/logs/$PROGRAM.log | \ sed
's/^\([[:digit:]]\+\)\..*$/\1/');
# ...
echo SBU: $(($time_elapsed / $standard_sbu))
Now, I tried for an hour or two to figure out a way that would redirect
various outputs and inputs via anonymous pipes and produce both the effect
of filling $time_elapsed and also print out the output on the screen. I
failed. Which was really dissapointing because I think that a system like
Linux should be able to do that. But no matter, the
infomation we need is inside the log file, so you just make a second pass
over the log file when all is done and extract the result.
Maybe perl or tlc would be able to do this task better. The fact that
the shell is unable to do what it SHOULD be able to do is quite embarrasing
indeed.
--
Svi moji e-mailovi su kriptografski potpisani. Proverite ih.
All of my e-mails are cryptographically signed. Verify them.
--
You don't need an AI for a robot uprising.
Humans will do just fine.
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Hello,
Thanks for this. It adds to my script collection nicely.
In the examples I was trying unsuccessfully to modify, they had the time
wrapped in multiple ((( and it just did not work.
Also from the research I discovered that the bash inbuilt time function
has a more limited amount of functionality than the gnu time. Though from
looking we do not seem to install any other version of time. It is listed
as released in 1996 on the gnu time website.
I am sorry that you spent so long in trying to get it to work correctly
via the anonymous pipes.
I was just going around in circles trying various things.
Regards,
Christopher.
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Re: [blfs-dev] bash script
On Tue, June 3, 2014 4:59 am, Bruce Dubbs wrote: Aleksandar Kuktin wrote: # ... yada yada yada ... } 21 | tee $SDIR/logs/$PROGRAM.log time_elapsed=$(grep 'Elapsed Time' $SDIR/logs/$PROGRAM.log | \ sed 's/^\([[:digit:]]\+\)\..*$/\1/'); # ... echo SBU: $(($time_elapsed / $standard_sbu)) But no matter, the infomation we need is inside the log file, so you just make a second pass over the log file when all is done and extract the result. Right. In my case, I happen to know that the line I need is the last one in the current log, so I use buildtime=`tail -n1 $log|cut -f1 -d ` Note that the time output honors the TIMEFORMAT environment variable, so I am using TIMEFORMAT=%1R Elapsed Time - $TITLE, so I get for output something like '123.4 Elapsed Time - some title' I then use bc to do the math and that gets a bit more accuracy, which helps especially for the packages that are less than 1 SBU. sbu=`echo scale=3; $buildtime / $base_sbu | bc` -- Bruce -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page Hello Bruce, Thanks for this. I think that I am going to be using your way of doing it for the gnome book. I did not know that there would be multiple replies to this, so I do feel bad that some may not get used. Though I can always try out some of the others, just to see if there are any differences in the calculations. It is really weird that the times do vary between runs of the script. Regards, Christopher. -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] bash script
On Tue, 3 Jun 2014 11:07:24 +1200 Christopher Gregory m...@pc-networking-services.com wrote: Also from the research I discovered that the bash inbuilt time function has a more limited amount of functionality than the gnu time. Though from looking we do not seem to install any other version of time. It is listed as released in 1996 on the gnu time website. Yeah, LFS mentions it (or mentioned it) in one or two places... I am sorry that you spent so long in trying to get it to work correctly via the anonymous pipes. In reality, I was sharpening my teeth on something I wanted to try for a long time now. You just provided me with motivation. ;) -- Svi moji e-mailovi su kriptografski potpisani. Proverite ih. All of my e-mails are cryptographically signed. Verify them. -- You don't need an AI for a robot uprising. Humans will do just fine. signature.asc Description: PGP signature -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
