Hi Thomas, Since you are available for suggestions.... :-) Perhaps it is a good idea to use the iso country codes from [1] to define a language. autowiki.sh be would fetch the belgium wikipedia in that case.
Kind regards, Ed [1] http://www.iso.org/iso/country_codes/iso_3166_code_lists/english_country_names_and_code_elements.htm On Tue, 2010-02-09 at 08:28 +0100, Thomas HOCEDEZ wrote: > (This message was first send on the wrong tree, sorry) > > Hi wikireaders, > > As I told before, my automated script is ready !And this script is > totally "cronable" :! > It manage every step, from downloading Wikipedia archive in your > language to the installation. > > it is available here : http://freerunner.daily.free.fr/files/autowiki.sh > Just modify with your ftp parameters (for ftp'ing images) & mail (to be > adverted on each step). then launch > > $ autowiki.sh fr > > for french Wikipedia > > $ autowiki be > > for Biello-russian Wikipedia > > > Some of you would say that it also could check if the wikireader folder > exists & install or upgrade source from the git, well, I let you some > work to do ;-) > > > I'm still available for any suggestion or anything else. > > The code is following. > > #!/bin/sh > # Automation script for rendering a Wikipedia image for the FR > # Written by AstHrO / openmoko-fr.org / [email protected] > # V 1.0 > # Todo > # param1 : language (fr,en,be,lu....) > > # ========================= > # Ftp host configuration : > > ftp_host="" > ftp_login="" > ftp_passd="" > mail="" > > # Folder where the wikireader tools are installed > > wr_folder="/media/stocks/wikireader" > > # ========================= > > # Language extension of the WP (fr,en, nl ...) > if [[ "$1" = "" ]]; then > lang="fr" > else > lang=$1 > fi > > > # You don't have to change following stuff, but you can ... > cfile="${lang}wiki-latest-pages-articles.xml.bz2" > ufile="${lang}wiki-latest-pages-articles.xml" > mail_msg="message.txt" > sep="==============================================================" > > echo "---" > $mail_msg > > # Going to the working folder : > cd $wr_folder > > # cleaning old treatments : > echo $sep > echo "Step 1 : CLeaning..." > make clean &>/dev/null > > # downloading interesting WP : > echo $sep > echo "Step 2 : Downloading dump ..." > rm $cfile &>null > wget http://download.wikimedia.org/${lang}wiki/latest/$cfile > ls -l $cfile | mail -s "[WR] Wikipedia dump downloaded" "$mail" < $mail_msg > > # uncompressing : > echo $sep > echo "Step 3 : Uncompressing ..." > rm $ufile &>/dev/null > bzip2 -d ./$cfile 2> log.txt &1> /dev/null > > #creating some folders : > mkdir work &>/dev/null > mkdir image &>/dev/null > > echo $sep > echo "Step 4 : Indexing Articles..." > # Creating index of articles : > make index XML_FILES="$ufile" DESTDIR=image WORKDIR=work 2> log.txt 1> > /dev/null > > # Parsing : (30>60 sec / 1000) > echo $sep > echo "Step 5 : Parsing Articles ..." > make parse -j3 XML_FILES="$ufile" DESTDIR=image WORKDIR=work 2>log.txt > #mail -s "[WR] Parsing of $ufile complete !" "$mail" < $mail_msg > > # Rendering the file : (60>200 sec / 1000) > echo $sep > echo "Step 6 : Rendering ..." > make render -j3 XML_FILES="$ufile" DESTDIR=image WORKDIR=work 2>log.txt > #mail -s "[WR] Rendering of $ufile complete !" "$mail" < $mail_msg > > echo $sep > echo "Step 7 : Finalizing ..." > # Combining articles indexes (few seconds) > make combine -j3 DESTDIR=image WORKDIR=work > $mail_msg > > # Generating a Hash (last few seconds dude !) > make hash -j3 DESTDIR=image WORKDIR=work >> $mail_msg > > # Going to output folder > cd image > echo $sep > echo "Step 8 : Compressing files..." > # Compressing data files > tar -cvf wr_${lang}_$(date '+%d-%m-%Y').tar.gz pedia*.* > > # a little HASH to be sure ... > shasum wr_${lang}_$(date '+%d-%m-%Y').tar.gz > sha_${lang}.txt > #mail -s "[WR] Hash of your file..." "$mail" < $mail_msg > echo $sep > echo "Step 9 : Let's FTP all this !" > # Let's FTP all this stuff > ftp -n << EOF > open $ftp_host > user $ftp_login $ftp_passd > binary > put wr_${lang}_$(date '+%d-%m-%Y').tar.gz > EOF > echo $sep > echo "Step 10 : Enjoy !" > #mail -s "[WR] All done, WP image ready to use !" "$mail" > > # That's it you can now send a mail to your friends. > > > > _______________________________________________ > Openmoko community mailing list > [email protected] > http://lists.openmoko.org/mailman/listinfo/community > -- > Thomas HOCEDEZ > _______________________________________________ > Openmoko community mailing list > [email protected] > http://lists.openmoko.org/mailman/listinfo/community _______________________________________________ Openmoko community mailing list [email protected] http://lists.openmoko.org/mailman/listinfo/community
