Am Montag, 7. August 2006 08:12 schrieb Pascal Bleser:
> houghi wrote:
> >
> > Yes, some thought needs to go into things. However due to the
> > openness op /usr/src/packages/RPMS I do not think it is a good idea
> > anymore.
No, using /usr/src/packages/RPMS directly is a pretty bad idea, building
packages as root either, thus there's no way around build stuff as user
and moving it to repo as root, AFAICT.
> >> Maybe a daemon, based on inotify, that triggers the rebuild
> >> everytime the repo changes, would be a smart solution to this
> >> problem..
> >
> > If I would have the knowledge on how to write such a deamon, I
> > would. Unfortunatly I am unable to do so. :-(
>
> Actually that should be quite easy to do with a shell script, using
> inotify-tools [1] [2]:
> ---8<------------------------------------------------------------
> #!/bin/sh
> RPMDIRS=/usr/src/packages/RPMS/*
> REPODIR=/usr/src/packages/RPMS
> CACHEDIR=/usr/src/packages/RPMS/.cache
>
> inotifywait -m -eclose $RPMDIRS | while read event; do
> createrepo -q --cachedir="$CACHEDIR" "$REPODIR" >/dev/null
> done
> ---8<------------------------------------------------------------
Very cool, Pascal.
> Would just need to make it somewhat smarter and only run createrepo
> if there hasn't been any write after a given amount of seconds (or
> minutes), because if rpmbuild creates several RPMS, it will trigger
> createrepo immediately although rpmbuild isn't finished with writing
> all of them.
Yes, that would be necessary. Will look into this as soon as I've
upgraded the kernel on my repo server (still running 9.3, hoping the
best for 10.2..), but this is planned anyway for several reasons.
> [1] http://inotify-tools.sourceforge.net/
> [2] SUSE RPMs available in my repo:
> http://linux01.gwdg.de/~pbleser/rpm-navigation.php?cat=System/inotify
>-tools (2.2 is built and underway)
>
> >> I've a patch pending to createrepo to speed this up considerably
> >> again, if the repo didn't change between invocations.
>
> What is that patch doing ? :)
Compare the ctime of the rpms with that of the repo meta data. See:
https://lists.dulug.duke.edu/pipermail/rpm-metadata/2006-July/000663.html
for a patch to createrepo-0.4.4 (the changelog patch will not apply, but
you can ignore that..)
Unfortunately, Seth Vidal and Paul Nasrat aren't the fastest guys on the
planet when it comes to _bug fixes_ from externals. Now imagine, how
they handle enhancement patches..
The checkts option has one well known downside ATM: it doesn't handle
the situation, when only packages got removed. Will add ctime check of
the directory as well, but that leads to spurious runs on rsynced
repos..
> Also, consider using createrepo --cache
Ahem, you meant --cachedir. That's what the -c option is doing as well,
but you're right, better use long options on ML discussions..
Pete
--- genpkgmetadata.py.orig 2006-07-19 19:39:07.042852436 +0200
+++ genpkgmetadata.py 2006-07-19 21:29:53.893998466 +0200
@@ -55,6 +55,8 @@ def usage(retval=1):
(<filename> relative to directory-of-packages)
-v, --verbose = run verbosely
-c, --cachedir <dir> = specify which dir to use for the checksum cache
+ -C, --checkts = don't generate repo metadata, if their ctimes are newer
+ than the rpm ctimes.
-h, --help = show this help
-V, --version = output version
-p, --pretty = output xml files in pretty format.
@@ -95,6 +97,17 @@ class MetaDataGenerator:
return filelist
+ def checkTimeStamps(self, directory):
+ if self.cmds['checkts']:
+ files = self.getFileList(self.cmds['basedir'], directory, '.rpm', [])
+ files = self.trimRpms(files)
+ for f in files:
+ fn = os.path.join(self.cmds['basedir'], directory, f)
+ if not os.path.exists(fn):
+ errorprint(_('cannot get to file: %s') % fn)
+ if os.path.getctime(fn) > self.cmds['mdtimestamp']:
+ return False
+ return True
def trimRpms(self, files):
badrpms = []
@@ -348,17 +361,19 @@ def parseArgs(args):
cmds['cachedir'] = None
cmds['basedir'] = os.getcwd()
cmds['cache'] = False
+ cmds['checkts'] = False
+ cmds['mdtimestamp'] = 0
cmds['split'] = False
cmds['outputdir'] = ""
cmds['file-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*', '^\/usr\/lib\/sendmail$']
cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*']
try:
- gopts, argsleft = getopt.getopt(args, 'phqVvng:s:x:u:c:o:', ['help', 'exclude=',
+ gopts, argsleft = getopt.getopt(args, 'phqVvng:s:x:u:c:o:C', ['help', 'exclude=',
'quiet', 'verbose', 'cachedir=', 'basedir=',
'baseurl=', 'groupfile=', 'checksum=',
'version', 'pretty', 'split', 'outputdir=',
- 'noepoch'])
+ 'noepoch', 'checkts'])
except getopt.error, e:
errorprint(_('Options Error: %s.') % e)
usage()
@@ -416,6 +431,8 @@ def parseArgs(args):
elif arg in ['-c', '--cachedir']:
cmds['cache'] = True
cmds['cachedir'] = a
+ elif arg in ['-C', '--checkts']:
+ cmds['checkts'] = True
elif arg == '--basedir':
cmds['basedir'] = a
elif arg in ['-o','--outputdir']:
@@ -427,6 +444,10 @@ def parseArgs(args):
errorprint(_('Options Error: %s') % e)
usage()
+ if cmds['split'] and cmds['checkts']:
+ errorprint(_('--split and --checkts options are mutually exclusive'))
+ sys.exit(1)
+
directory = directories[0]
# Fix paths
directory = os.path.normpath(directory)
@@ -507,13 +528,21 @@ def main(args):
if not os.access(filepath, os.W_OK):
errorprint(_('error in must be able to write to metadata files:\n -> %s') % filepath)
usage()
-
+ if cmds['checkts']:
+ ts = os.path.getctime(filepath)
+ if ts > cmds['mdtimestamp']:
+ cmds['mdtimestamp'] = ts
+
if cmds['split']:
cmds['basedir'] = oldbase
mdgen = SplitMetaDataGenerator(cmds)
mdgen.doPkgMetadata(directories)
else:
mdgen = MetaDataGenerator(cmds)
+ if cmds['checkts'] and mdgen.checkTimeStamps(directory):
+ if cmds['verbose']:
+ print _('repo is up to date')
+ sys.exit(0)
mdgen.doPkgMetadata(directory)
mdgen.doRepoMetadata()
--- ChangeLog.orig 2006-07-19 21:12:39.546793958 +0200
+++ ChangeLog 2006-07-19 21:29:48.365194764 +0200
@@ -1,3 +1,10 @@
+2006-07-19 21:03 [EMAIL PROTECTED]
+
+ * genpkgmetadata.py, docs/createrepo.8:
+ -C, --checkts option added to avoid metadata generation, if ctime
+ filestamps are up to date. It's currently mutually exclusive with
+ the --split option.
+
2006-06-28 17:03 [EMAIL PROTECTED]
* dumpMetadata.py:
--- docs/createrepo.8.orig 2006-07-19 21:12:54.095275300 +0200
+++ docs/createrepo.8 2006-07-19 21:29:45.396300173 +0200
@@ -32,6 +32,12 @@ cache of checksums of packages in the re
createrepo over the same repository of files that do not have a complete
change out of all packages this decreases the processing time dramatically.
.br
+.IP "\fB\-C --checkts\fP"
+Don't generate repo metadata, if their timestamps are newer than its rpms.
+This option decreases the processing time drastically again, if you happen
+to run it on an unmodified repo, but it is (currently) mutual exclusive
+with the --split option.
+.br
.IP "\fB\--split\fP"
Run in split media mode. Rather than pass a single directory, take a set of
directories corresponding to different volumes in a media set.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
