Hi all, other work for the bittorrent transport...
In this patch I've added to BOEL binaries the standard bittorrent python client "freezed" as a normal ELF binary, as suggested by Bernard in a previous chat with him.
This is a very important issue, because with the python interpreter the BOEL binaries took a lot of memory in the clients and a lot of time to be deployed... This is because I've always prefered the Bernard approach using libBT... :), but now also with this approach the size of the binaries is strongly reduced (from 20-30MB to 6.7MB!).
To create the bittorrent binaries from the python scripts, I've used cx_Freeze (http://starship.python.net/crew/atuining/cx_Freeze/).
Moreover I've implemented a new heuristic for a client to decide when to stop seeding and go forward with the installation. In particular parsing the bittorrent-console output is possible to get the upload rate (= how many KB/s are pushed to the other peers). Looking periodically at this value is possible to decide that under a certain minimum threshold the client/seeder is no more useful, and it can proceed to the final steps of the installation (untar the image, run systemconfigurator, etc.).
I've done some test and it seems to work very nice! Unfortunately at the moment I've only 1 real client for the tests... so I had to do some tricks to simulate more clients on another installed machine running more bittorrent-console on it... :-( this is very bad, anyway I suppose it should be a simple but quite reliable approximation... so who as 100 clients to do some better tests? :-))
Comments are welcome... Best regards, -Andrea
Index: patches/bittorrent.freezehack.patch =================================================================== --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 ++++ bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++ import re ++ import encodings ++ import encodings.ascii ++ import encodings.base64_codec ++ import encodings.big5hkscs ++ import encodings.big5 ++ import encodings.bz2_codec ++ import encodings.charmap ++ import encodings.cp037 ++ import encodings.cp1006 ++ import encodings.cp1026 ++ import encodings.cp1140 ++ import encodings.cp1250 ++ import encodings.cp1251 ++ import encodings.cp1252 ++ import encodings.cp1253 ++ import encodings.cp1254 ++ import encodings.cp1255 ++ import encodings.cp1256 ++ import encodings.cp1257 ++ import encodings.cp1258 ++ import encodings.cp424 ++ import encodings.cp437 ++ import encodings.cp500 ++ import encodings.cp737 ++ import encodings.cp775 ++ import encodings.cp850 ++ import encodings.cp852 ++ import encodings.cp855 ++ import encodings.cp856 ++ import encodings.cp857 ++ import encodings.cp860 ++ import encodings.cp861 ++ import encodings.cp862 ++ import encodings.cp863 ++ import encodings.cp864 ++ import encodings.cp865 ++ import encodings.cp866 ++ import encodings.cp869 ++ import encodings.cp874 ++ import encodings.cp875 ++ import encodings.cp932 ++ import encodings.cp949 ++ import encodings.cp950 ++ import encodings.euc_jis_2004 ++ import encodings.euc_jisx0213 ++ import encodings.euc_jp ++ import encodings.euc_kr ++ import encodings.gb18030 ++ import encodings.gb2312 ++ import encodings.gbk ++ import encodings.hex_codec ++ import encodings.hp_roman8 ++ import encodings.hz ++ import encodings.idna ++ import encodings.iso2022_jp_1 ++ import encodings.iso2022_jp_2004 ++ import encodings.iso2022_jp_2 ++ import encodings.iso2022_jp_3 ++ import encodings.iso2022_jp_ext ++ import encodings.iso2022_jp ++ import encodings.iso2022_kr ++ import encodings.iso8859_10 ++ import encodings.iso8859_11 ++ import encodings.iso8859_13 ++ import encodings.iso8859_14 ++ import encodings.iso8859_15 ++ import encodings.iso8859_16 ++ import encodings.iso8859_1 ++ import encodings.iso8859_2 ++ import encodings.iso8859_3 ++ import encodings.iso8859_4 ++ import encodings.iso8859_5 ++ import encodings.iso8859_6 ++ import encodings.iso8859_7 ++ import encodings.iso8859_8 ++ import encodings.iso8859_9 ++ import encodings.johab ++ import encodings.koi8_r ++ import encodings.koi8_u ++ import encodings.latin_1 ++ import encodings.mac_cyrillic ++ import encodings.mac_greek ++ import encodings.mac_iceland ++ import encodings.mac_latin2 ++ import encodings.mac_roman ++ import encodings.mac_turkish ++ import encodings.mbcs ++ import encodings.palmos ++ import encodings.ptcp154 ++ import encodings.punycode ++ import encodings.quopri_codec ++ import encodings.raw_unicode_escape ++ import encodings.rot_13 ++ import encodings.shift_jis_2004 ++ import encodings.shift_jis ++ import encodings.shift_jisx0213 ++ import encodings.string_escape ++ import encodings.tis_620 ++ import encodings.undefined ++ import encodings.unicode_escape ++ import encodings.unicode_internal ++ import encodings.utf_16_be ++ import encodings.utf_16_le ++ import encodings.utf_16 ++ import encodings.utf_7 ++ import encodings.utf_8 ++ import encodings.uu_codec ++ import encodings.zlib_codec ++ import cStringIO ++ import gc ++ import xml.dom ++except: ++ pass ++ + from BitTorrent.platform import install_translation + install_translation() + Index: sbin/si_startbttrack =================================================================== --- sbin/si_startbttrack (revision 0) +++ sbin/si_startbttrack (revision 0) @@ -0,0 +1,139 @@ +#!/usr/bin/perl -w +# +# "SystemImager" +# +# Copyright (C) 2005 Andrea Righi <[EMAIL PROTECTED]> + +use lib "USR_PREFIX/lib/systemimager/perl"; +use strict; +use Getopt::Long; +use vars qw($config $VERSION); +use constant DEFAULT_TRACKER_PORT => 6969; + +my $VERSION = "SYSTEMIMAGER_VERSION_STRING"; +my $program_name = "si_startbttrack"; +my $version_info = << "EOF"; +$program_name (part of SystemImager) v$VERSION + +Copyright (C) 1999-2001 Andrea Righi <[EMAIL PROTECTED]> +This is free software; see the source for copying conditions. There is NO +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +EOF + +my $get_help = "\n Try \"--help\" for more options."; + +my $help_info = $version_info . <<"EOF"; + +Usage: $program_name --image_server <Hostname|IP> --image <IMAGE> + [OPTION]... + +Options: (options can be presented in any order and may be abbreviated) + --help Display this output. + + --version Display version and copyright information. + + --image_server HOSTNAME | IP + Specify the hostname or ip of the image server. + + --image IMAGENAME + Specify the name of the image to distribute. + + --tracker_port PORT Specify an optional tracker port (default 6969). + + --update_image Update image tar file. + + --quiet Run image server seeder in background. + +Download, report bugs, and make suggestions at: +http://systemimager.org/ + +EOF + +my ($help, $version, $quiet, $update_image, $image_server, $image_name, $tracker_port); +GetOptions( + "help" => \$help, + "version" => \$version, + "quiet" => \$quiet, + "update_image" => \$update_image, + "image_server=s" => \$image_server, + "tracker_port=s" => \$tracker_port, + "image=s" => \$image_name, +) or die "$help_info"; + +### BEGIN evaluate commad line options ### +if ($help) { + print "$help_info"; + exit(0); +} + +if ($version) { + print "$version_info"; + exit(0); +} + +unless ($image_server) { + die "\n$program_name: --image_server HOSTNAME | IP must be specified.\n$get_help\n\n"; +} + +unless ($image_name) { + die "\n$program_name: --image IMAGENAME must be specified.\n$get_help\n\n"; +} + +unless ($tracker_port) { + $tracker_port = DEFAULT_TRACKER_PORT; +} + +my $IMAGE_DIR = "/var/lib/systemimager/images"; +my $SCRIPTS_DIR = "/var/lib/systemimager/scripts"; + +unless (-d "$IMAGE_DIR/$image_name") { + die "\n$program_name --image $image_name: not a valid image!.\n$get_help\n\n"; +} + +# Prepare tar file. +print "Preparing tar file for $image_name...\n"; +if ($update_image) { + unlink("$IMAGE_DIR/$image_name.tar"); +} +unless (-f "$IMAGE_DIR/$image_name.tar") { + system "cd $IMAGE_DIR/$image_name && tar -cf $IMAGE_DIR/$image_name.tar ."; + if ($?) { + die "error: cannot create image tar file!\n"; + } +} +print "done.\n"; + +# Prepare torrent file. +print "Preparing torrent file for $image_name...\n"; +if ($update_image) { + unlink("$SCRIPTS_DIR/$image_name.tar.torrent"); +} +unless (-f "$SCRIPTS_DIR/$image_name.tar.torrent") { + system "maketorrent-console --target $SCRIPTS_DIR/$image_name.tar.torrent http://$image_server:$tracker_port/announce $IMAGE_DIR/$image_name.tar"; + if ($?) { + die "error: cannot create torrent file!\n"; + } +} +print "done.\n"; + +# Start tracker. +print "Starting tracker...\n"; +unlink('/tmp/dstate', '/var/log/systemimager/tracker.log'); +system "bittorrent-tracker --port $tracker_port --dfile /tmp/dstate --logfile /var/log/systemimager/tracker.log &"; +if ($?) { + die "error: cannot start tracker!\n"; +} +print "done.\n"; + +print "Starting first seeder...\n"; +if ($quiet) { + $quiet = '>/dev/null 2>&1 &'; +} else { + $quiet = ''; +} +system "cd $IMAGE_DIR && bittorrent-console --no_start_trackerless_client --max_upload_rate 0 $SCRIPTS_DIR/$image_name.tar.torrent $quiet"; +print "done.\n"; + +# Well done. +exit(0); + Property changes on: sbin/si_startbttrack ___________________________________________________________________ Name: svn:executable + * Index: make.d/bittorrent.rul =================================================================== --- make.d/bittorrent.rul (revision 0) +++ make.d/bittorrent.rul (revision 0) @@ -0,0 +1,36 @@ +# +# 2005-12-28 Andrea Righi +# - Bittorrent binaries +# + + +BITTORRENT_VERSION := 4.3.4 +BITTORRENT_DIR := BitTorrent-$(BITTORRENT_VERSION) +BITTORRENT_TARBALL := BitTorrent-$(BITTORRENT_VERSION).tar.gz +BITTORRENT_URL := http://www.bittorrent.com/dl/$(BITTORRENT_TARBALL) +#BITTORRENT_URL := http://download.systemimager.org/pub/bittorrent/$(BITTORRENT_TARBALL) +BITTORRENT_BINARY := $(SRC_DIR)/$(BITTORRENT_DIR)/bittorrent.py +BITTORRENT_PATCHES := $(shell ls $(PATCH_DIR)/bittorrent.*.patch 2>/dev/null | sort) + + +PHONY += bittorrent + +bittorrent: $(BITTORRENT_BINARY) + +$(BITTORRENT_BINARY): $(SRC_DIR)/$(BITTORRENT_TARBALL) + rm -rf $(SRC_DIR)/$(BITTORRENT_DIR) + cd $(SRC_DIR) && tar -xvzf $(BITTORRENT_TARBALL) + cd $(SRC_DIR)/$(BITTORRENT_DIR) && \ + patch -p0 < $(BITTORRENT_PATCHES) + +$(SRC_DIR)/$(BITTORRENT_TARBALL): + [ -d $(SRC_DIR) ] || mkdir -p $(SRC_DIR) + $(GETSOURCE) $(BITTORRENT_URL) $(SRC_DIR) + +ALL_SOURCE += $(SRC_DIR)/$(BITTORRENT_TARBALL) + +PHONY += bittorrent_clean +bittorrent_clean: + rm -rf $(SRC_DIR)/$(BITTORRENT_DIR) + +# /* vi: set noet ts=4: */ Index: make.d/cx_freeze.rul =================================================================== --- make.d/cx_freeze.rul (revision 0) +++ make.d/cx_freeze.rul (revision 0) @@ -0,0 +1,34 @@ +# +# 2005-12-28 Andrea Righi +# - Tool to freeze python script into binary +# + + +CX_FREEZE_VERSION := 3.0.2 +CX_FREEZE_DIR := cx_Freeze-$(CX_FREEZE_VERSION) +CX_FREEZE_TARBALL := cx_Freeze-$(CX_FREEZE_VERSION)-source.tgz +CX_FREEZE_URL := http://unc.dl.sourceforge.net/sourceforge/cx-freeze/$(CX_FREEZE_TARBALL) +#CX_FREEZE_URL := http://download.systemimager.org/pub/cx-freeze/$(CX_FREEZE_TARBALL) +CX_FREEZE_BINARY := $(SRC_DIR)/$(CX_FREEZE_DIR)/FreezePython + +PHONY += cx_freeze + +cx_freeze: $(CX_FREEZE_BINARY) + +$(CX_FREEZE_BINARY): $(SRC_DIR)/$(CX_FREEZE_TARBALL) + rm -rf $(SRC_DIR)/$(CX_FREEZE_DIR) + cd $(SRC_DIR) && tar -xvzf $(CX_FREEZE_TARBALL) + cd $(SRC_DIR)/$(CX_FREEZE_DIR) && $(PYTHON) MakeFrozenBases.py && \ + $(PYTHON) FreezePython.py --no-copy-deps FreezePython.py + +$(SRC_DIR)/$(CX_FREEZE_TARBALL): + [ -d $(SRC_DIR) ] || mkdir -p $(SRC_DIR) + $(GETSOURCE) $(CX_FREEZE_URL) $(SRC_DIR) + +ALL_SOURCE += $(SRC_DIR)/$(CX_FREEZE_TARBALL) + +PHONY += cx_freeze_clean +cx_freeze_clean: + rm -rf $(SRC_DIR)/$(CX_FREEZE_DIR) + +# /* vi: set noet ts=4: */ Index: initrd_source/skel/etc/init.d/functions =================================================================== --- initrd_source/skel/etc/init.d/functions (revision 3394) +++ initrd_source/skel/etc/init.d/functions (working copy) @@ -193,6 +193,8 @@ echo "MONITOR_SERVER=$MONITOR_SERVER" >> /tmp/variables.txt echo "MONITOR_PORT=$MONITOR_PORT" >> /tmp/variables.txt echo "MONITOR_CONSOLE=$MONITOR_CONSOLE" >> /tmp/variables.txt + + echo "BITTORRENT_STAGING=$BITTORRENT_STAGING" >> /tmp/variables.txt } # ################################################################################ @@ -1418,7 +1420,11 @@ # Evaluate image size. logmsg "Evaluating image size." - IMAGESIZE=`rsync -av $IMAGESERVER::$IMAGENAME | grep "total size" | sed -e "s/total size is \([0-9]*\).*/\1/"` + if [ -z $BITTORRENT_STAGING ]; then + IMAGESIZE=`rsync -av $IMAGESERVER::$IMAGENAME | grep "total size" | sed -e "s/total size is \([0-9]*\).*/\1/"` + else + IMAGESIZE=`torrentinfo-console /scripts/$IMAGENAME.tar.torrent | sed -ne "s/file size\.*: \([0-9]*\) .*$/\1/p"` + fi IMAGESIZE=`expr $IMAGESIZE / 1024` # Evaluate disks size. Index: etc/autoinstallscript.template =================================================================== --- etc/autoinstallscript.template (revision 3394) +++ etc/autoinstallscript.template (working copy) @@ -216,6 +216,44 @@ if [ ! -z $MONITOR_SERVER ]; then stop_report_task 100 fi +elif [ ! -z $BITTORRENT_STAGING ]; then + # Bittorrent log file + bittorrent_log=/tmp/bittorrent.log + # Time to poll bittorrent events + bittorrent_polling_time=30 + # Minimum upload rate threshold (KB/s), if lesser stop seeding + bittorrent_upload_min=1 + # Download image from peers + logmsg "Start downloading image using torrent ${IMAGENAME}.tar.torrent" + cd ${BITTORRENT_STAGING} && bittorrent-console --no_start_trackerless_client --max_upload_rate 0 --display_interval 1 --bind ${IPADDR} /scripts/${IMAGENAME}.tar.torrent > $bittorrent_log & + pid=$! + while :; do + status=`grep 'percent done:' $bittorrent_log | sed -ne '$p' | sed 's/percent done: *//'` + [ -z "$status" ] && status=0.0 + logmsg "percent done: $status %" + if [ "$status" = "100.0" ]; then + # Sleep until upload rate reaches the minimum threshold + while :; do + upload_rate=`grep 'upload rate:' $bittorrent_log | sed -ne '$p' | sed 's/upload rate: *\([0-9\.]*\) .*$/\1/'` + logmsg "upload rate: $upload_rate KB/s" + [ `echo $upload_rate \< $bittorrent_upload_min | bc` = 1 ] && break + sleep $bittorrent_polling_time + done + logmsg "Stop seeding" + kill -9 $pid + unset bittorrent_log pid upload_rate + break + fi + sleep $bittorrent_polling_time + done + unset bittorrent_polling_time + if [ ! -z $MONITOR_SERVER ]; then + stop_report_task 100 + fi + # Extract image. + logmsg "Extracting image ${IMAGENAME}.tar ..." + (cd /a/ && tar -xvf ${BITTORRENT_STAGING}/${IMAGENAME}.tar) || shellout + rm -f ${BITTORRENT_STAGING}/${IMAGENAME}.tar else # Use rsync if [ $NO_LISTING ]; then Index: Makefile =================================================================== --- Makefile (revision 3394) +++ Makefile (working copy) @@ -164,7 +164,7 @@ PXE_CONF_DEST = $(ETC)/systemimager/pxelinux.cfg BINARIES := si_mkautoinstallcd si_mkautoinstalldiskette si_mkbootmedia -SBINARIES := si_addclients si_cpimage si_getimage si_mkdhcpserver si_mkdhcpstatic si_mkautoinstallscript si_mkbootserver si_mvimage si_pushupdate si_rmimage si_mkrsyncd_conf si_mkclientnetboot si_netbootmond si_imagemanip si_mkbootpackage si_monitor si_monitortk +SBINARIES := si_addclients si_cpimage si_getimage si_mkdhcpserver si_mkdhcpstatic si_mkautoinstallscript si_mkbootserver si_mvimage si_pushupdate si_rmimage si_mkrsyncd_conf si_mkclientnetboot si_netbootmond si_imagemanip si_mkbootpackage si_monitor si_monitortk si_startbttrack CLIENT_SBINARIES := si_updateclient si_prepareclient COMMON_BINARIES = si_lsimage @@ -314,6 +314,9 @@ $(SI_INSTALL) -d -m 755 $(FLAMETHROWER_STATE_DIR) + # Install server-side BitTorrent. + cd $(SRC_DIR)/$(BITTORRENT_DIR) && $(PYTHON) setup.py install --prefix $(PREFIX) + # install client-only files .PHONY: install_client install_client: install_client_man install_client_libs @@ -497,6 +500,8 @@ $(OPENSSH_BINARIES) \ $(OPENSSH_CONF_FILES) \ $(LVM_BINARY) \ + $(CX_FREEZE_BINARY) \ + $(BITTORRENT_BINARY) \ $(SRC_DIR)/modules_build-stamp # # Put binaries in the boel_binaries_tarball... @@ -535,6 +540,10 @@ # cd $(BOEL_BINARIES_DIR)/sbin && $(foreach binary,$(shell cat $(SRC_DIR)/$(LVM_DIR)/tools/.commands),ln -s -f lvm $(binary) && ) /bin/true + # Bittorrent installation. + $(CX_FREEZE_BINARY) --install-dir=$(BOEL_BINARIES_DIR)/lib/bittorrent --exclude-modules=gobject,pango,gtk $(SRC_DIR)/$(BITTORRENT_DIR)/torrentinfo-console.py + $(CX_FREEZE_BINARY) --install-dir=$(BOEL_BINARIES_DIR)/lib/bittorrent --exclude-modules=gobject,pango,gtk $(SRC_DIR)/$(BITTORRENT_DIR)/bittorrent-console.py + mkdir -m 755 -p $(BOEL_BINARIES_DIR)/lib test ! -d /lib64 || mkdir -m 755 -p $(BOEL_BINARIES_DIR)/lib64 @@ -591,6 +600,12 @@ test ! -d /lib64 || TGTLIBDIR=lib64 ; \ cd $(BOEL_BINARIES_DIR) \ && $(PYTHON) $(TOPDIR)/initrd_source/mklibs -L $(SRC_DIR)/$(PARTED_DIR)/libparted/.libs:$(SRC_DIR)/$(DISCOVER_DIR)/lib/.libs:$(SRC_DIR)/$(DEVMAPPER_DIR)/lib/ioctl:$(SRC_DIR)/$(E2FSPROGS_DIR)/lib:/lib64:/usr/lib64:/usr/kerberos/lib64:/lib:/usr/lib:/usr/kerberos/lib -v -d $$TGTLIBDIR bin/* sbin/* + + # Create the bittorrent links after mklibs: cx_Freeze already + # provides the opportune libraries needed by the binaries. -AR- + cd $(BOEL_BINARIES_DIR)/bin && ln -s ../lib/bittorrent/bittorrent-console + cd $(BOEL_BINARIES_DIR)/bin && ln -s ../lib/bittorrent/torrentinfo-console + # # Include other files required by openssh that apparently aren't # picked up by mklibs for some reason. -BEF-
