I have a very easy fix for this - you want to limit the rate at which
read or write the data. As tge gzip/tar process will then have lots of
breathing time for CPU/disk intensive operations:
There is a very nice pipe viewer/rate limiter called "pv"
1. So write your command so it uses stdout to write (or read) the file:
gzip -c big_file > small_file.gz
2. Now use "pv" to view the rate at which it runs
gzip -c big_file | pv > small_file.gz
This will show output like:-
3.33MB 0:00:02 [ 1.7MB/s] [ <=>
3. Now use "pv" to rate limit
gzip -c big_file | pv -L 100k > small_file.gz
You get output like this
292kB 0:00:03 [ 98kB/s] [ <=>
As gzip is only allowed to write 100kB per second it continually yields
CPU cycles back to the kernel.
I grabbed pv from http://www.ivarch.com/programs/pv.shtml
Martin
Martin Visser ,CISSP
Network and Security Consultant
Consulting & Integration
Technology Solutions Group - HP Services
3 Richardson Place
North Ryde, Sydney NSW 2113, Australia
Phone: +61-2-9022-1670
Mobile: +61-411-254-513
Fax: +61-2-9022-1800
E-mail: martin.visserAThp.com
This email (including any attachments) is intended only for the use of
the individual or entity named above and may contain information that is
confidential, proprietary or privileged. If you are not the intended
recipient, please notify HP immediately by return email and then delete
the email, destroy any printed copy and do not disclose or use the
information in it.
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Benno
> Sent: Wednesday, 9 February 2005 11:40 AM
> To: Steve Kowalik
> Cc: [email protected]; David
> Subject: Re: [SLUG] a nice question
>
> On Wed Feb 09, 2005 at 11:36:13 +1100, Steve Kowalik wrote:
> >On Wed, 9 Feb 2005 11:12:11 +1100, David uttered
> >> I guess this is the point of the question. Is there a way of
> >> prioritising so that the effect of tar running is minimised.
> >>
> >Sure. Start tar and then renice the process to 20 (it will only run
> >when nothing else wants to), man renice for more information.
>
> But nice only effects the CPU usage, and doesn't do anything
> with the amount of buffer cache being thrashed, or the memory
> bus being killed by disk transfers or the increased seek
> times for any other applications, etc, etc, etc.
>
> Fixing this would be an interested project for aspiring
> kernel hackers :
>
> Benno
> --
> SLUG - Sydney Linux User's Group Mailing List -
> http://slug.org.au/ Subscription info and FAQs:
> http://slug.org.au/faq/mailinglists.html
>
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
