Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-28 Thread Paul de Vrieze 
On Thursday 26 October 2006 00:57, Alec Warner wrote:
 Caleb Cushing wrote:
  reporting additions of new programs aren't feasible? or are you
  referring to version updates and package bumps and such

 Reporting removals will be done by treecleaners once I have it implemented.

 Reporting additions may require some cvs foo on lark; such as new
 directories in  ${PORTDIR}/$CAT/

 Someone needs to implement the foo; however.

(Not on cvs, but on a normal tree, but maybe works on cvs. There is a sanity 
check by checking that a dir contains at least 1 ebuild)

= start =
OLDPKGS=/var/cache/gwn/oldpkgs
NEWPKGS=`mktemp -t`
find /usr/portage -mindepth 2 -maxdepth 2 |while read in
do
  EB=`echo $in/*.ebuild`
  if [ -n $EB ]; then
echo $in
  fi
done $NEWPKGS
#These are new packages
cat $OLDPKGS $OLDPKGS $NEWPKGS|sort |uniq -u

#Old packages can be retrieved almost similarly:
#cat $OLDPKGS $NEWPKGS $NEWPKGS|sort |uniq -u

mv $NEWPKGS $OLDPKGS
= end =

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: [EMAIL PROTECTED]
Homepage: http://www.devrieze.net


pgp2RvKjuIfSk.pgp
Description: PGP signature

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-28 Thread Ciaran McCreesh 
On Sat, 28 Oct 2006 20:42:14 +0200 Paul de Vrieze [EMAIL PROTECTED]
wrote:
|  Someone needs to implement the foo; however.
| 
| (Not on cvs, but on a normal tree, but maybe works on cvs. There is a
| sanity check by checking that a dir contains at least 1 ebuild)

If the foo were that simple, someone would have done it long ago. It
needs to take into account updates/.

-- 
Ciaran McCreesh
Mail: ciaranm at ciaranm.org
Web : http://ciaranm.org/
as-needed is broken : http://ciaranm.org/show_post.pl?post_id=13



signature.asc
Description: PGP signature

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-28 Thread Alec Warner 

Paul de Vrieze wrote:

On Thursday 26 October 2006 00:57, Alec Warner wrote:

Caleb Cushing wrote:

reporting additions of new programs aren't feasible? or are you
referring to version updates and package bumps and such

Reporting removals will be done by treecleaners once I have it implemented.

Reporting additions may require some cvs foo on lark; such as new
directories in  ${PORTDIR}/$CAT/

Someone needs to implement the foo; however.


(Not on cvs, but on a normal tree, but maybe works on cvs. There is a sanity 
check by checking that a dir contains at least 1 ebuild)


= start =
OLDPKGS=/var/cache/gwn/oldpkgs
NEWPKGS=`mktemp -t`
find /usr/portage -mindepth 2 -maxdepth 2 |while read in
do
  EB=`echo $in/*.ebuild`
  if [ -n $EB ]; then
echo $in
  fi
done $NEWPKGS
#These are new packages
cat $OLDPKGS $OLDPKGS $NEWPKGS|sort |uniq -u

#Old packages can be retrieved almost similarly:
#cat $OLDPKGS $NEWPKGS $NEWPKGS|sort |uniq -u

mv $NEWPKGS $OLDPKGS
= end =

Paul



I've got some foo that can run on lark and figure out new and removed 
packages; as well as do it given date ranges.  It uses the cvs history 
file.  My foo basically looks at new and removed Changelogs in 
gentoo-x86.  It excludes profiles/ (there are Changelogs in there). 
I'll post it a bit later; I have a few bits of it to fix.


It doesn't cover updates/.  I don't think that corner case is easily 
covered.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-28 Thread Yuri Vasilevski 
Hi,

On Sat, 28 Oct 2006 19:58:09 +0100
Ciaran McCreesh [EMAIL PROTECTED] wrote:

 On Sat, 28 Oct 2006 20:42:14 +0200 Paul de Vrieze [EMAIL PROTECTED]
 wrote:
 |  Someone needs to implement the foo; however.
 | 
 | (Not on cvs, but on a normal tree, but maybe works on cvs. There is
 a | sanity check by checking that a dir contains at least 1 ebuild)
 
 If the foo were that simple, someone would have done it long ago. It
 needs to take into account updates/.

I have sent my script directly to wolf31o2 some days ago thinking that
the exact technical details can considered spam for the majority
subscribed on the list, but for completeness I'm attaching in in this
e-mail.

It works comparing two different portage snapshots that it can also
download.

Best,
Yuri.

gen_newpkg_table.sh
Description: application/shellscript

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-28 Thread Alec Warner 

Yuri Vasilevski wrote:

Hi,

On Sat, 28 Oct 2006 19:58:09 +0100
Ciaran McCreesh [EMAIL PROTECTED] wrote:


On Sat, 28 Oct 2006 20:42:14 +0200 Paul de Vrieze [EMAIL PROTECTED]
wrote:
|  Someone needs to implement the foo; however.
| 
| (Not on cvs, but on a normal tree, but maybe works on cvs. There is

a | sanity check by checking that a dir contains at least 1 ebuild)

If the foo were that simple, someone would have done it long ago. It
needs to take into account updates/.


I have sent my script directly to wolf31o2 some days ago thinking that
the exact technical details can considered spam for the majority
subscribed on the list, but for completeness I'm attaching in in this
e-mail.

It works comparing two different portage snapshots that it can also
download.

Best,
Yuri.


I was planning to have this cron'd on lark (cvs.gentoo.org) and run 
against the cvs history file.
#!/usr/bin/env python
# Authored by Alec Warner [EMAIL PROTECTED]
# Released under the GPL Version 2
# Copyright Gentoo Foundation 2006

# Changelog: Initial release 2006/10/27

doc = 
# Purpose: This script analyzes the cvs history file in an attempt to locate package
# additions and removals.  It takes 3 arguments; two of which are optional.  It needs
# the path to the history file to read.  If a start_date is not provided it will read
# the entire file and match any addition/removal.  If you provide a start date it will
# only match thins that are after that start_date.  If you provide an end date you can
# find matches over date ranges.  If an end date is not provided it defaults to now()


import sys, os, re, time, datetime

new_package = re.compile(^A(.*)\|.*gentoo-x86\/(.*)\/(.*)\|.*\|ChangeLog$)
removed_package = re.compile(^R(.*)\|.*gentoo-x86\/(.*)\/(.*)\|.*\|ChangeLog$)

class record(object):
	def __init__(self, who, date, cp):
		
		Who is a string
		date is a unix timestamp
		cp is a category/package
		
		self.who = who
		self.date = datetime.datetime.fromtimestamp( date ) 
		self.package = cp

	def __str__( self ):
		return Package %s was removed by %s on %s % (self.package, self.who, self.date)

def main():
	if (len(sys.argv)  2):
		usage()
		sys.exit(1)

	args = sys.argv[1:]
	history_file = args[0]

	if len(args) = 2: 
		start_date = args[1]
		start_date = time.strptime( start_date, %d/%m/%Y)
		start_date = time.mktime( start_date )
	else:
		start_date = 0 # no real start date then.

	if len(args) = 3:
		end_date = args[2]
		end_date = time.strptime( end_date, %d/%m/%Y)
		end_date = time.mktime( end_date )
	else:
		end_date = time.time()

	try:
		lines = open( history_file ).readlines()
	except (IOError, OSError), e:
		print Failed to open History file!
		raise e

	removals = []
	adds = []
	for line in lines:
		match = new_package.match( line )
		if match:
			t = match.groups()[0]
			split = t.split(|)
			t = split[0]
			who = split[1]
			try:
t = int(t, 16)
			except e:
print Failed to convert hex timestamp to integer
raise e

			if t  end_date and t  start_date:
rec = record( who, t, match.groups()[1] + / + match.groups()[2] )
adds.append( rec )
continue
			else:
continue # date out of range
		match = removed_package.match( line )

		if match:
			t = match.groups()[0]
			split = t.split(|)
			t = split[0]
			who = split[1]
			try:
t = int(t, 16)
			except e:
print Failed to convert hex timestamp to integer
raise e
			if t  end_date and t  start_date:
rec = record( who, t, match.groups()[1] + / + match.groups()[2] )
removals.append( rec )
continue
			else:
continue # date out of range
	print Removed Packages:
	for pkg in removals:
		print pkg

	print Added Packages:
	for pkg in adds:
		print pkg
	print
	print Done.

def usage():
	print sys.argv[0] +  history file start date [end date]
	print start date should be in dd/mm/ format
	print end date is not required and defaults to 'now'
	print doc

if __name__ == __main__:
	main()

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-28 Thread Marius Mauch 
On Sat, 28 Oct 2006 17:09:10 -0400
Alec Warner [EMAIL PROTECTED] wrote:

 It doesn't cover updates/.  I don't think that corner case is easily 
 covered.

Ehm, if you have a list of CP entries that were added/removed and
access to the update files it shouldn't be all that complicated.

Just made up pseudocode:

for x in $raw_added_cps; do
grep ^move [^ ]* $x\$ $current_update_file ||
$real_added_cps=$real_added_cps $x
done

Hardest part is probably to select $current_update_file correctly.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-25 Thread Diego 'Flameeyes' Pettenò 
On Wednesday 25 October 2006 21:56, Yuri Vasilevski wrote:
 After seeing Upcoming package removals, for couple of weeks now, in
 GWN I'm beginning to think that I would like to see also a list of new
 packages added to portage next to the list of packages to be removed.
http://packages.gentoo.org/

-- 
Diego Flameeyes Pettenò - http://farragut.flameeyes.is-a-geek.org/
Gentoo/Alt lead, Gentoo/FreeBSD, Video, Sound, ALSA, PAM, KDE, CJK, Ruby ...


pgpW9VJdUOkQ7.pgp
Description: PGP signature

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-25 Thread Chris Gianelloni 
On Wed, 2006-10-25 at 14:56 -0500, Yuri Vasilevski wrote:
 If you think this is a good idea, I'd be glad to write some scripts for
 this.

If you write all the code, I'll run it, but I'm not taking my time to
track down all of the additions.  The current removals is done by hand
by some volunteers for the GWN.  There's nothing automatic about it.
Someone else already suggested this to us, but I explained to him that
unless someone had some way to automate it that it simply wasn't
feasible to do by hand.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation


signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-25 Thread Caleb Cushing 

reporting additions of new programs aren't feasible? or are you
referring to version updates and package bumps and such
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-25 Thread Chris Gianelloni 
On Wed, 2006-10-25 at 16:25 -0400, Caleb Cushing wrote:
 reporting additions of new programs aren't feasible? or are you
 referring to version updates and package bumps and such

None of it is feasible if I'm left to do it by hand.  I have much better
things to do (like actually add new packages) than try to keep track of
what everyone else is doing.  The GWN takes a significant amount of time
as it is to put together.  Any process that isn't automated is pretty
much out of the question due to time constraints.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation


signature.asc
Description: This is a digitally signed message part

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

2006-10-25 Thread Alec Warner 

Caleb Cushing wrote:

reporting additions of new programs aren't feasible? or are you
referring to version updates and package bumps and such


Reporting removals will be done by treecleaners once I have it implemented.

Reporting additions may require some cvs foo on lark; such as new 
directories in  ${PORTDIR}/$CAT/


Someone needs to implement the foo; however.
--
gentoo-dev@gentoo.org mailing list