Author: sebb
Date: Wed Oct 14 23:28:15 2015
New Revision: 1708714
URL: http://svn.apache.org/viewvc?rev=1708714&view=rev
Log:
Let's give parsecommitteeinfo a go
Added:
comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py
- copied, changed from r1708707,
comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py
Removed:
comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py
Modified:
comdev/projects.apache.org/STRUCTURE.txt
comdev/projects.apache.org/scripts/README.txt
Modified: comdev/projects.apache.org/STRUCTURE.txt
URL:
http://svn.apache.org/viewvc/comdev/projects.apache.org/STRUCTURE.txt?rev=1708714&r1=1708713&r2=1708714&view=diff
==============================================================================
--- comdev/projects.apache.org/STRUCTURE.txt (original)
+++ comdev/projects.apache.org/STRUCTURE.txt Wed Oct 14 23:28:15 2015
@@ -33,9 +33,10 @@ Suggested cron setup:
scripts/cronjobs/podlings.py - daily
scripts/cronjobs/countaccounts.py - weekly
scripts/cronjobs/parsereleases.py - daily
+ scripts/cronjob/parsecommitteeinfo.py - daily
Stuff to run manually when needed:
- scripts/import/parsecommittees.py
+ scripts/import/parsecommittees.py - obsolete
scripts/import/parseprojects.py
Webserver required:
@@ -57,6 +58,7 @@ crontab -l -u www-data:
00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3
parsechairs.py
00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3
countaccounts.py
00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3
parsereleases.py
+00 01 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3
parsecommitteeinfo.py
10 4 * * * cd /var/www/projects.apache.org/site/json && ( svn status | awk
'/^\? / {print $2}' | xargs -r svn add )
Modified: comdev/projects.apache.org/scripts/README.txt
URL:
http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/README.txt?rev=1708714&r1=1708713&r2=1708714&view=diff
==============================================================================
--- comdev/projects.apache.org/scripts/README.txt (original)
+++ comdev/projects.apache.org/scripts/README.txt Wed Oct 14 23:28:15 2015
@@ -11,6 +11,19 @@ various sources:
in: data/cache/committee-info.json (from whimsy, via committee_info module)
out: site/json/foundation/chairs.json - (used by reporter.a.o only)
+- parsecommitteeinfo.py: Parses committee-info.json to detect new and retired
committees
+ and imports PMC data (RDF) from PMC data files
+ in: site/json/foundation/committees.json
+ + site/json/foundation/committees-retired.json
+ + https://whimsy.apache.org/public/committee-info.json (via
committee_info.py)
+ + data/committees.xml - list of where to find PMC description RDF files
+ + data/committees/*.rdf - local PMC description RDF files
+ out: site/json/foundation/committees.json (updated)
+ + site/json/foundation/committees-retired.json (updated)
+ + site/json/foundation/pmcs.json - (used by reporter.a.o only)
+ + site/doap/{committeeId}/pmc-doap.rdf - this is an exact copy of the
file listed in data/committees.xml
+ + site/doap/{committeeId}/pmc.rdf - TODO: what purpose does this serve?
+
- parsecommitters.py: Fetches and parses the committer (LDAP) list via
people.apache.org.
in: http://people.apache.org/committer-index.html
@@ -30,20 +43,12 @@ various sources:
out: json/foundation/releases.json
+ json/foundation/releases-files.json
+
2. importing data (import)
- parsecommittees.py: Parses committee-info.txt to detect new and retired
committees and imports PMC data (RDF) from
PMC data files
- in: site/json/foundation/committees.json
- + site/json/foundation/committees-retired.json
- + data/board/committee-info.txt
(https://svn.apache.org/repos/private/committers/board/committee-info.txt)
- + data/committees.xml - list of where to find PMC description RDF files
- + data/committees/*.rdf - local PMC description RDF files
- out: site/json/foundation/committees.json (updated)
- + site/json/foundation/committees-retired.json (updated)
- + site/json/foundation/pmcs.json - (used by reporter.a.o only)
- + site/doap/{committeeId}/pmc-doap.rdf - this is an exact copy of the
file listed in data/committees.xml
- + site/doap/{committeeId}/pmc.rdf - TODO: what purpose does this serve?
+ No longer needed, use parsecommitteeinfo instead
- parseprojects.py: Parses existing projects RDF(DOAP) files and turns them
into JSON objects.
in: data/projects.xml + projects' DOAP files
@@ -53,7 +58,7 @@ various sources:
NOTICE: what prevents import scripts to be added to cron?
1. parse committees.py requires committee-info.txt, which is not available on
project-vm (require authentication)
- TODO: Whimsy now supplies a JSON version of CI
+ Whimsy now supplies a JSON version of CI
2. both scripts not only update files but sometimes need to add new files (new
committees or new projects) or move
(projects going to Attic or retired committees)
TODO: any reason why scripts should not do this automatically?
Copied: comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py (from
r1708707, comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py)
URL:
http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py?p2=comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py&p1=comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py&r1=1708707&r2=1708714&rev=1708714&view=diff
==============================================================================
--- comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py
(original)
+++ comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py Wed Oct
14 23:28:15 2015
@@ -17,9 +17,6 @@ import committee_info
"""
- THIS IS A PRELIMINARY VERSION
- DO NOT USE YET
- =============================
Reads:
../../site/json/foundation/people.json
../../data/committees.xml
