Re: Fun Scripting Problem
On 2013-02-13 12:27, Tim Daneliuk wrote: > I know how to do this in Python, but I really want to do it in > straight Bourne shell. I have some ideas, but I thought I'd > give you folks a crack at this Big Fun: > > a) You have a directory of files - say they're logs - generated > at nondeterministic intervals. You may get more than one a day, > more than one a month, none, or hundreds. > > b) To conserve space, you want to keep the last file generated >in any given month (the archive goes back for an unspecified >number of years), and delete all the files generated prior to >that last file in that same month. > > c) Bonus points if the problem is solved generally for either files >or directories generated as described above. > > These are not actually logs, and no, I don't think logrotate can > do this ... or can it? You can try using stat/date to print the month in which each file was modified, e.g: freefall:/home/keramida$ stat -f '%m %N' prs.tar.bz2 1359910210 prs.tar.bz2 freefall:/home/keramida$ date -f '%s' -j '1359910210' '+%Y-%m' 2013-02 Having the mtime of the file in seconds since the epoch (1359910210) should be easy to sort through, e.g. you can find the last modification time with a 'sort -n | tail -1' pipe: freefall:/home/keramida/w/doc-head/el_GR.ISO8859-7/share$ touch xml/glossary.ent freefall:/home/keramida/w/doc-head/el_GR.ISO8859-7/share$ find xml -exec stat -f '%m %N' {} + 1360060289 xml 1360060288 xml/navibar.l10n.ent 1360060288 xml/trademarks.ent 1360060288 xml/libcommon.xsl 1360060288 xml/header.l10n.ent => 1360942284 xml/glossary.ent 1360060289 xml/freebsd.dsl 1360060289 xml/teams.ent 1360060289 xml/freebsd.ent 1360060289 xml/l10n.ent 1360060289 xml/mailing-lists.ent 1360060289 xml/newsgroups.ent 1360060289 xml/translators.ent 1360060289 xml/catalog.xml 1360060289 xml/entities.ent 1360060289 xml/catalog 1360060289 xml/urls.ent freefall:/home/keramida/w/doc-head/el_GR.ISO8859-7/share$ find xml -exec stat -f '%m %N' {} + | sort -n | tail -1 1360942284 xml/glossary.ent Then you can convert the epoch-based times to '%Y-%m' timestamps in a loop, e.g.: freefall:/home/keramida/w/doc-head/el_GR.ISO8859-7/share$ find xml \ > -exec stat -f '%m %N' {} + | while read mtime fname ; do > echo ${mtime} $( date -f '%s' -j "${mtime}" '+%Y-%m' ) ${fname} > done 1360060289 2013-02 xml 1360060288 2013-02 xml/navibar.l10n.ent 1360060288 2013-02 xml/trademarks.ent 1360060288 2013-02 xml/libcommon.xsl 1360060288 2013-02 xml/header.l10n.ent 1360942284 2013-02 xml/glossary.ent 1360060289 2013-02 xml/freebsd.dsl 1360060289 2013-02 xml/teams.ent 1360060289 2013-02 xml/freebsd.ent 1360060289 2013-02 xml/l10n.ent 1360060289 2013-02 xml/mailing-lists.ent 1360060289 2013-02 xml/newsgroups.ent 1360060289 2013-02 xml/translators.ent 1360060289 2013-02 xml/catalog.xml 1360060289 2013-02 xml/entities.ent 1360060289 2013-02 xml/catalog 1360060289 2013-02 xml/urls.ent Having the mtime in seconds as the first column is still conducive to sorting / tail: freefall:/home/keramida/w/doc-head/el_GR.ISO8859-7/share$ find xml \ > -exec stat -f '%m %N' {} + | while read mtime fname ; do > echo ${mtime} $( date -f '%s' -j "${mtime}" '+%Y-%m' ) ${fname}; > done | sort -n | tail -1 1360942284 2013-02 xml/glossary.ent >From that point it should be trivial to select all files whose timestamp is smaller than or equal to 1360942284 - one month of seconds. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Fun Scripting Problem
On Wed, Feb 13, 2013 at 03:13:06PM -0600, Robert Bonomi wrote: > > here's a one-liner: > rm ` \ > stat -f "%SB %B %N" * \ > | sort -k5nr \ > | cut -c1-7,17-20,32- \ > | awk 'BEGIN {a="";b=0;c=0} $1==a && $2==b && $3=c {print > $4;}{a=$1;b=$2;c=$3}' \ I'm never comfortable calling something like that a "one-liner". If it runs over 80 columns of width, that (to me) doesn't really qualify as a one-liner. -- Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ] signature.asc Description: Digital signature
Re: Fun Scripting Problem
On 02/13/2013 03:13 PM, Robert Bonomi wrote: Date: Wed, 13 Feb 2013 12:27:31 -0600 From: Tim Daneliuk Subject: Fun Scripting Problem I know how to do this in Python, but I really want to do it in straight Bourne shell. I have some ideas, but I thought I'd give you folks a crack at this Big Fun: a) You have a directory of files - say they're logs - generated at nondeterministic intervals. You may get more than one a day, more than one a month, none, or hundreds. b) To conserve space, you want to keep the last file generated in any given month (the archive goes back for an unspecified number of years), and delete all the files generated prior to that last file in that same month. c) Bonus points if the problem is solved generally for either files or directories generated as described above. These are not actually logs, and no, I don't think logrotate can do this ... or can it? here's a one-liner: rm ` \ stat -f "%SB %B %N" * \ | sort -k5nr \ | cut -c1-7,17-20,32- \ | awk 'BEGIN {a="";b=0;c=0} $1==a && $2==b && $3=c {print $4;}{a=$1;b=$2;c=$3}' \ ` This selects on creation date. change the B (both of them) in the stat call to use a different timestamp Thanks to all that took the time. Interesting responses. It will be fun to cook up my own version. -- Tim Daneliuk tun...@tundraware.com PGP Key: http://www.tundraware.com/PGP/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Fun Scripting Problem
>> On Wed, 13 Feb 2013 12:53:32 -0600, >> Tim Daneliuk said: T> The only way to determine the date of the file is by looking at its stat T> info. There is nothing the file name or content that could be used to T> infer this. Being a pedantic twit, I interpreted "stat info" to mean "info you can get from something that calls stat()". The script below runs on BSD, Linux, or Solaris if you have GNU find installed. Season to taste. -- Karl Vogel I don't speak for the USAF or my company Mom was so overprotective, she only let us play "Rock, Paper". --- #!/bin/ksh # usage: keepfiles DIR export PATH=/usr/local/bin:/bin:/usr/bin tag=${0##*/} here=$(pwd) tmp="$here/keep$$" # Sanity check. die () { echo "$tag: FATAL: $@" >&2; exit 1; } # Don't repeat yourself. Use undocumented option %Ts to # avoid dealing with fractional seconds. listdir () { find . -type f -printf "%TY%Tm%Td %Ts|%p\n"; } case "$#" in 0) die "need a directory" ;; *) work="$1" ;; esac # List files by date generated. mkdir $tmp|| die "$tmp: mkdir failed" test -d $work || die "$work: not a directory" cd $work || die "cd $work failed" months=$(listdir | cut -c1-6 | sort -u) for mon in $months do listdir | # Get the files to check, grep "^$mon" | # ... look for each month, sort -n | # ... sort by modtime, cut -d'|' -f2 | # ... get the path, tail -1 | # ... last one's oldest, sed -e 's!^\(.*\)!mv \1 '$tmp'!' | # ... write a mv command, sh # ... and run it. done # Clean up. cd $here echo "Keep the stuff in $tmp" exit 0 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Fun Scripting Problem
> Date: Wed, 13 Feb 2013 12:27:31 -0600 > From: Tim Daneliuk > Subject: Fun Scripting Problem > > I know how to do this in Python, but I really want to do it in > straight Bourne shell. I have some ideas, but I thought I'd > give you folks a crack at this Big Fun: > > a) You have a directory of files - say they're logs - generated > at nondeterministic intervals. You may get more than one a day, > more than one a month, none, or hundreds. > > b) To conserve space, you want to keep the last file generated > in any given month (the archive goes back for an unspecified > number of years), and delete all the files generated prior to > that last file in that same month. > > c) Bonus points if the problem is solved generally for either files > or directories generated as described above. > > These are not actually logs, and no, I don't think logrotate can > do this ... or can it? here's a one-liner: rm ` \ stat -f "%SB %B %N" * \ | sort -k5nr \ | cut -c1-7,17-20,32- \ | awk 'BEGIN {a="";b=0;c=0} $1==a && $2==b && $3=c {print $4;}{a=$1;b=$2;c=$3}' \ ` This selects on creation date. change the B (both of them) in the stat call to use a different timestamp ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Fun Scripting Problem
Tim Daneliuk writes: > On 02/13/2013 12:38 PM, Teske, Devin wrote: >> (apologies for top-post) >> >> As tempted as I am, I think newsyslog(8) may be what you want. >> >> Missing information in your post is how you intend to timestamp the >> files -- by filename? by content? If by-content, then is it a good >> assumption that the data is one entry per-line? ... and if-so, is >> the timestamp in that line? These are all questions that would be >> needed to script what you're asking for (not that I'm volunteering >> or anything like that). >> > > The only way to determine the date of the file is by looking at its > stat info. There is nothing the file name or content that could > be used to infer this. Well, you can use stat to output the year, month, timestamp, and file name in a fixed format for all of the files, sort them, and then cycle through the list, deleting every file that has a year and month that are the same as the following one in the list. The looping can be done with sh "read" or with sed. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Fun Scripting Problem
On 02/13/2013 12:38 PM, Teske, Devin wrote: (apologies for top-post) As tempted as I am, I think newsyslog(8) may be what you want. Missing information in your post is how you intend to timestamp the files -- by filename? by content? If by-content, then is it a good assumption that the data is one entry per-line? ... and if-so, is the timestamp in that line? These are all questions that would be needed to script what you're asking for (not that I'm volunteering or anything like that). The only way to determine the date of the file is by looking at its stat info. There is nothing the file name or content that could be used to infer this. -- --- Tim Daneliuk ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
RE: Fun Scripting Problem
(apologies for top-post) As tempted as I am, I think newsyslog(8) may be what you want. Missing information in your post is how you intend to timestamp the files -- by filename? by content? If by-content, then is it a good assumption that the data is one entry per-line? ... and if-so, is the timestamp in that line? These are all questions that would be needed to script what you're asking for (not that I'm volunteering or anything like that). -- Devin From: owner-freebsd-questi...@freebsd.org [owner-freebsd-questi...@freebsd.org] on behalf of Tim Daneliuk [tun...@tundraware.com] Sent: Wednesday, February 13, 2013 10:27 AM To: FreeBSD Mailing List Subject: Fun Scripting Problem I know how to do this in Python, but I really want to do it in straight Bourne shell. I have some ideas, but I thought I'd give you folks a crack at this Big Fun: a) You have a directory of files - say they're logs - generated at nondeterministic intervals. You may get more than one a day, more than one a month, none, or hundreds. b) To conserve space, you want to keep the last file generated in any given month (the archive goes back for an unspecified number of years), and delete all the files generated prior to that last file in that same month. c) Bonus points if the problem is solved generally for either files or directories generated as described above. These are not actually logs, and no, I don't think logrotate can do this ... or can it? -- --- Tim Daneliuk ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"