Re: [Sisuite-devel] Re: Bit-Torrent transport
I know it... if I remember well this should be the first bittorrent client, in the authors of the project is reported also Bram Cohen. This should be very very similar to the client we're using (there is also the same interface and same commands available, maybe only the "backend" changes). Anyway it could be interesting to do some tests also with bittornado, and it should be very easy to integrate it in systemimager, I mean it should be very similar to the current python bt client... I'll do that ASAP... Cheers, -Andrea Bernard Li wrote: > Hey Andrea: > > I have heard good things about BitTornado as a BT client, perhaps you > can do some performance tests too with it when you get a chance: > > http://www.bittornado.com/ > > Cheers, > > Bernard > --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 ___ Sisuite-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sisuite-devel
RE: [Sisuite-devel] Re: Bit-Torrent transport
Title: Re: [Sisuite-devel] Re: Bit-Torrent transport Hey Andrea: I have heard good things about BitTornado as a BT client, perhaps you can do some performance tests too with it when you get a chance: http://www.bittornado.com/ Cheers, Bernard From: Andrea Righi [mailto:[EMAIL PROTECTED]Sent: Wed 08/02/2006 14:25To: [EMAIL PROTECTED]Cc: Bernard Li; [EMAIL PROTECTED]; Brian Elliott Finley; [EMAIL PROTECTED]; SIS DevelSubject: Re: [Sisuite-devel] Re: Bit-Torrent transport Last changes:- splitted scripts and torrents: added the parameterAUTOINSTALL_TORRENT_DIR in /etc/systemimager/systemimager.conf to definethe place where torrents resides on the image server (default:/var/lib/systemimager/torrents/)- added the function bittorrent_get_file() that performs all the genericcommands to download a file by bittorrentRegards,-Andrea
Re: [Sisuite-devel] Re: Bit-Torrent transport
Last changes: - splitted scripts and torrents: added the parameter AUTOINSTALL_TORRENT_DIR in /etc/systemimager/systemimager.conf to define the place where torrents resides on the image server (default: /var/lib/systemimager/torrents/) - added the function bittorrent_get_file() that performs all the generic commands to download a file by bittorrent Regards, -Andrea Index: patches/bittorrent.freezehack.patch === --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++import re ++import encodings ++import encodings.ascii ++import encodings.base64_codec ++import encodings.big5hkscs ++import encodings.big5 ++import encodings.bz2_codec ++import encodings.charmap ++import encodings.cp037 ++import encodings.cp1006 ++import encodings.cp1026 ++import encodings.cp1140 ++import encodings.cp1250 ++import encodings.cp1251 ++import encodings.cp1252 ++import encodings.cp1253 ++import encodings.cp1254 ++import encodings.cp1255 ++import encodings.cp1256 ++import encodings.cp1257 ++import encodings.cp1258 ++import encodings.cp424 ++import encodings.cp437 ++import encodings.cp500 ++import encodings.cp737 ++import encodings.cp775 ++import encodings.cp850 ++import encodings.cp852 ++import encodings.cp855 ++import encodings.cp856 ++import encodings.cp857 ++import encodings.cp860 ++import encodings.cp861 ++import encodings.cp862 ++import encodings.cp863 ++import encodings.cp864 ++import encodings.cp865 ++import encodings.cp866 ++import encodings.cp869 ++import encodings.cp874 ++import encodings.cp875 ++import encodings.cp932 ++import encodings.cp949 ++import encodings.cp950 ++import encodings.euc_jis_2004 ++import encodings.euc_jisx0213 ++import encodings.euc_jp ++import encodings.euc_kr ++import encodings.gb18030 ++import encodings.gb2312 ++import encodings.gbk ++import encodings.hex_codec ++import encodings.hp_roman8 ++import encodings.hz ++import encodings.idna ++import encodings.iso2022_jp_1 ++import encodings.iso2022_jp_2004 ++import encodings.iso2022_jp_2 ++import encodings.iso2022_jp_3 ++import encodings.iso2022_jp_ext ++import encodings.iso2022_jp ++import encodings.iso2022_kr ++import encodings.iso8859_10 ++import encodings.iso8859_11 ++import encodings.iso8859_13 ++import encodings.iso8859_14 ++import encodings.iso8859_15 ++import encodings.iso8859_16 ++import encodings.iso8859_1 ++import encodings.iso8859_2 ++import encodings.iso8859_3 ++import encodings.iso8859_4 ++import encodings.iso8859_5 ++import encodings.iso8859_6 ++import encodings.iso8859_7 ++import encodings.iso8859_8 ++import encodings.iso8859_9 ++import encodings.johab ++import encodings.koi8_r ++import encodings.koi8_u ++import encodings.latin_1 ++import encodings.mac_cyrillic ++import encodings.mac_greek ++import encodings.mac_iceland ++import encodings.mac_latin2 ++import encodings.mac_roman ++import encodings.mac_turkish ++import encodings.mbcs ++import encodings.palmos ++import encodings.ptcp154 ++import encodings.punycode ++import encodings.quopri_codec ++import encodings.raw_unicode_escape ++import encodings.rot_13 ++import encodings.shift_jis_2004 ++import encodings.shift_jis ++import encodings.shift_jisx0213 ++import encodings.string_escape ++import encodings.tis_620 ++import encodings.undefined ++import encodings.unicode_escape ++import encodings.unicode_internal ++import encodings.utf_16_be ++import encodings.utf_16_le ++import encodings.utf_16 ++import encodings.utf_7 ++import encodings.utf_8 ++import encodings.uu_codec ++import encodings.zlib_codec ++import cStringIO ++import gc ++import xml.dom ++except: ++pass ++ + from BitTorrent.platform import install_translation + install_translation() + Index: sbin/si_installbtimage === --- sbin/si_installbtimage (revision 0) +++ sbin/si_installbtimage (revision 0) @@ -0,0 +1,299 @@ +#!/usr/bin/perl -w +# +# "SystemImager" +# +# Copyright (C) 2005 Andrea Righi <[EMAIL PROTECTED]> + +use lib "USR_PREFIX/lib/systemimager/perl"; +use strict; +use POSIX; +use Getopt::Long; +use SystemImager::Config; +use vars qw($config $VERSION); + +my $config_dir = "/etc/systemimager"; + +my $VERSION = "SYSTEMIMAGER_VERSION_STRING"; +my $program_name = "si_installbtimage"; +my
Re: [Sisuite-devel] Re: Bit-Torrent transport
Bernard Li wrote: >> Moreover looking at some bittorrent papers it seems that there are not >> any improvements in synchronizing the clients installation (I mean to >> synchronize the image download from all clients). So IMO there are no >> meaninful advantages integrating the bittorrent ransport in >> flametrhrower. Do you see any other advantages other than >> performances? >> otherwise I think it'll be better to use bittorrent as a standalone >> transport (like rsync), since in this way it'll be easier for >> a user to >> configure and setup the bittorrent stuff, in fact flamethrower needs a >> lot of preparations and setups to be used... > > I discussed this with Brian previously and it was agreed that if you are > imaging 1000 nodes, using bittorrent for boel_binaries, image, scripts, > etc. will help. Of course if you are just deploying ~10 nodes the > overhead may be too great, and may not be worth the while to use the > bittorrent transport. OK this is a good motivation, even if now that the bittorrent client is in the initrd I already use bittorrent to download both boel_binaries and image.. only the scripts are downloaded via rsync (but because inside this directory I put also the .torrent files and these must be downloaded using another protocols). A cleaner solution should be to create a separate dir also for the torrent, i.e. /var/lib/systemimager/torrents seems to be a good candidate... In this way we can download all in the same way as with flamethrower... > > I'd also like to add that what we might end up doing is add support for > stand-alone bittorrent transport (if we want to) but the option of > running bittorrent via flamethrower is also available... > Great! It shouldn't be too difficult IMO to include bt transport both standalone and inside flamethrower... for the first time I think it's the better solution, and this will help us to understand better which is the best approach. I like it! :) > Again, if we are able to run some tests, all these things will be more > apparent. > Yes.. I'm sure some tests will explain a lot of things... Cheers, -Andrea --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 ___ Sisuite-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sisuite-devel
RE: [Sisuite-devel] Re: Bit-Torrent transport
Hi Andrea: > - btget (libbt): max bandwidth 1.6MB/s, average 1.1MB/s > - BitTorrent client (freezed python scripts): max bandwidth 4.9 MB/s, > average 4.3MB/s Yeah, I don't understand why libbt performs so poorly... perhaps we should give libtorrent a shot. > Moreover looking at some bittorrent papers it seems that there are not > any improvements in synchronizing the clients installation (I mean to > synchronize the image download from all clients). So IMO there are no > meaninful advantages integrating the bittorrent ransport in > flametrhrower. Do you see any other advantages other than > performances? > otherwise I think it'll be better to use bittorrent as a standalone > transport (like rsync), since in this way it'll be easier for > a user to > configure and setup the bittorrent stuff, in fact flamethrower needs a > lot of preparations and setups to be used... I discussed this with Brian previously and it was agreed that if you are imaging 1000 nodes, using bittorrent for boel_binaries, image, scripts, etc. will help. Of course if you are just deploying ~10 nodes the overhead may be too great, and may not be worth the while to use the bittorrent transport. I'd also like to add that what we might end up doing is add support for stand-alone bittorrent transport (if we want to) but the option of running bittorrent via flamethrower is also available... Again, if we are able to run some tests, all these things will be more apparent. Cheers, Bernard --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642 ___ Sisuite-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sisuite-devel
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi all, I've been very busy in these last days, so my reply comes only now... Beside all, I've added some minor fixes and improvements in my bittorrent patch (attached), in particular I moved to the version 4.4.0, which seems to offer very better performances compared with the previous release. I did also additional tests comparing performances between libbt (using btget) and the standard python client. Results on a single client are followings: - btget (libbt): max bandwidth 1.6MB/s, average 1.1MB/s - BitTorrent client (freezed python scripts): max bandwidth 4.9 MB/s, average 4.3MB/s These values are reported directly by the bittorrent client on the image server, si_monitortk statistics differ for about a 10%, since it considers the blocks written to the disk to evaluate the transfer bandwidth and obviously there is a little overhead in space from the real bytes transmitted. As I said I got the values installing a single client, so they are not so meaningful... I hope in the next week I'll be able to repeat my tests with 6-7 clients... and so I hope I'll be able to give meaningful results... Moreover looking at some bittorrent papers it seems that there are not any improvements in synchronizing the clients installation (I mean to synchronize the image download from all clients). So IMO there are no meaninful advantages integrating the bittorrent ransport in flametrhrower. Do you see any other advantages other than performances? otherwise I think it'll be better to use bittorrent as a standalone transport (like rsync), since in this way it'll be easier for a user to configure and setup the bittorrent stuff, in fact flamethrower needs a lot of preparations and setups to be used... Best regards, -Andrea Index: patches/bittorrent.freezehack.patch === --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++import re ++import encodings ++import encodings.ascii ++import encodings.base64_codec ++import encodings.big5hkscs ++import encodings.big5 ++import encodings.bz2_codec ++import encodings.charmap ++import encodings.cp037 ++import encodings.cp1006 ++import encodings.cp1026 ++import encodings.cp1140 ++import encodings.cp1250 ++import encodings.cp1251 ++import encodings.cp1252 ++import encodings.cp1253 ++import encodings.cp1254 ++import encodings.cp1255 ++import encodings.cp1256 ++import encodings.cp1257 ++import encodings.cp1258 ++import encodings.cp424 ++import encodings.cp437 ++import encodings.cp500 ++import encodings.cp737 ++import encodings.cp775 ++import encodings.cp850 ++import encodings.cp852 ++import encodings.cp855 ++import encodings.cp856 ++import encodings.cp857 ++import encodings.cp860 ++import encodings.cp861 ++import encodings.cp862 ++import encodings.cp863 ++import encodings.cp864 ++import encodings.cp865 ++import encodings.cp866 ++import encodings.cp869 ++import encodings.cp874 ++import encodings.cp875 ++import encodings.cp932 ++import encodings.cp949 ++import encodings.cp950 ++import encodings.euc_jis_2004 ++import encodings.euc_jisx0213 ++import encodings.euc_jp ++import encodings.euc_kr ++import encodings.gb18030 ++import encodings.gb2312 ++import encodings.gbk ++import encodings.hex_codec ++import encodings.hp_roman8 ++import encodings.hz ++import encodings.idna ++import encodings.iso2022_jp_1 ++import encodings.iso2022_jp_2004 ++import encodings.iso2022_jp_2 ++import encodings.iso2022_jp_3 ++import encodings.iso2022_jp_ext ++import encodings.iso2022_jp ++import encodings.iso2022_kr ++import encodings.iso8859_10 ++import encodings.iso8859_11 ++import encodings.iso8859_13 ++import encodings.iso8859_14 ++import encodings.iso8859_15 ++import encodings.iso8859_16 ++import encodings.iso8859_1 ++import encodings.iso8859_2 ++import encodings.iso8859_3 ++import encodings.iso8859_4 ++import encodings.iso8859_5 ++import encodings.iso8859_6 ++import encodings.iso8859_7 ++import encodings.iso8859_8 ++import encodings.iso8859_9 ++import encodings.johab ++import encodings.koi8_r ++import encodings.koi8_u ++import encodings.latin_1 ++import encodings.mac_cyrillic ++import encodings.mac_greek ++import encodings.mac_iceland ++import encodings.mac_latin2 ++import encodings.mac_roman ++import encodings.mac_turkish ++import encodings.mbcs ++import encodings.palmos ++import encodings.ptcp154 ++import
Re: [Sisuite-devel] Re: Bit-Torrent transport
Restructured si_installbtimage to seed more images and BOEL binaries at the same time using a single seeder on the image server... Cheers, -Andrea Index: patches/bittorrent.freezehack.patch === --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++import re ++import encodings ++import encodings.ascii ++import encodings.base64_codec ++import encodings.big5hkscs ++import encodings.big5 ++import encodings.bz2_codec ++import encodings.charmap ++import encodings.cp037 ++import encodings.cp1006 ++import encodings.cp1026 ++import encodings.cp1140 ++import encodings.cp1250 ++import encodings.cp1251 ++import encodings.cp1252 ++import encodings.cp1253 ++import encodings.cp1254 ++import encodings.cp1255 ++import encodings.cp1256 ++import encodings.cp1257 ++import encodings.cp1258 ++import encodings.cp424 ++import encodings.cp437 ++import encodings.cp500 ++import encodings.cp737 ++import encodings.cp775 ++import encodings.cp850 ++import encodings.cp852 ++import encodings.cp855 ++import encodings.cp856 ++import encodings.cp857 ++import encodings.cp860 ++import encodings.cp861 ++import encodings.cp862 ++import encodings.cp863 ++import encodings.cp864 ++import encodings.cp865 ++import encodings.cp866 ++import encodings.cp869 ++import encodings.cp874 ++import encodings.cp875 ++import encodings.cp932 ++import encodings.cp949 ++import encodings.cp950 ++import encodings.euc_jis_2004 ++import encodings.euc_jisx0213 ++import encodings.euc_jp ++import encodings.euc_kr ++import encodings.gb18030 ++import encodings.gb2312 ++import encodings.gbk ++import encodings.hex_codec ++import encodings.hp_roman8 ++import encodings.hz ++import encodings.idna ++import encodings.iso2022_jp_1 ++import encodings.iso2022_jp_2004 ++import encodings.iso2022_jp_2 ++import encodings.iso2022_jp_3 ++import encodings.iso2022_jp_ext ++import encodings.iso2022_jp ++import encodings.iso2022_kr ++import encodings.iso8859_10 ++import encodings.iso8859_11 ++import encodings.iso8859_13 ++import encodings.iso8859_14 ++import encodings.iso8859_15 ++import encodings.iso8859_16 ++import encodings.iso8859_1 ++import encodings.iso8859_2 ++import encodings.iso8859_3 ++import encodings.iso8859_4 ++import encodings.iso8859_5 ++import encodings.iso8859_6 ++import encodings.iso8859_7 ++import encodings.iso8859_8 ++import encodings.iso8859_9 ++import encodings.johab ++import encodings.koi8_r ++import encodings.koi8_u ++import encodings.latin_1 ++import encodings.mac_cyrillic ++import encodings.mac_greek ++import encodings.mac_iceland ++import encodings.mac_latin2 ++import encodings.mac_roman ++import encodings.mac_turkish ++import encodings.mbcs ++import encodings.palmos ++import encodings.ptcp154 ++import encodings.punycode ++import encodings.quopri_codec ++import encodings.raw_unicode_escape ++import encodings.rot_13 ++import encodings.shift_jis_2004 ++import encodings.shift_jis ++import encodings.shift_jisx0213 ++import encodings.string_escape ++import encodings.tis_620 ++import encodings.undefined ++import encodings.unicode_escape ++import encodings.unicode_internal ++import encodings.utf_16_be ++import encodings.utf_16_le ++import encodings.utf_16 ++import encodings.utf_7 ++import encodings.utf_8 ++import encodings.uu_codec ++import encodings.zlib_codec ++import cStringIO ++import gc ++import xml.dom ++except: ++pass ++ + from BitTorrent.platform import install_translation + install_translation() + Index: sbin/si_installbtimage === --- sbin/si_installbtimage (revision 0) +++ sbin/si_installbtimage (revision 0) @@ -0,0 +1,288 @@ +#!/usr/bin/perl -w +# +# "SystemImager" +# +# Copyright (C) 2005 Andrea Righi <[EMAIL PROTECTED]> + +use lib "USR_PREFIX/lib/systemimager/perl"; +use strict; +use POSIX; +use Getopt::Long; +use SystemImager::Config; +use vars qw($config $VERSION); + +my $config_dir = "/etc/systemimager"; + +my $VERSION = "SYSTEMIMAGER_VERSION_STRING"; +my $program_name = "si_installbtimage"; +my $version_info = << "EOF"; +$program_name (part of SystemImager) v$VERSION + +Copyright (C) 1999-2001 Andrea Righi <[EMAIL PROTECTED]> +This is free software; see the source for copying conditions. There is NO +warranty; not
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi all, what's new: - now also the BOEL binaries are distributed via bittorrent - modified si_installbtimage to seed both the boel binaries and the tarball of the image (using the commands launchmay-console / launchmany-curses); I'm considering to do another update, so that the si_installbtimage will be able to distribute _all_ the image tarballs, binaries, etc. using a single seeder... - some fixes for the move of the bittorrent client into the initrd Moreover I've started to look at the Bernard patch, in particular regarding the flamethrower integration; tonight I'll try to do some tests with libbt client. Regards, -Andrea Index: patches/bittorrent.freezehack.patch === --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++import re ++import encodings ++import encodings.ascii ++import encodings.base64_codec ++import encodings.big5hkscs ++import encodings.big5 ++import encodings.bz2_codec ++import encodings.charmap ++import encodings.cp037 ++import encodings.cp1006 ++import encodings.cp1026 ++import encodings.cp1140 ++import encodings.cp1250 ++import encodings.cp1251 ++import encodings.cp1252 ++import encodings.cp1253 ++import encodings.cp1254 ++import encodings.cp1255 ++import encodings.cp1256 ++import encodings.cp1257 ++import encodings.cp1258 ++import encodings.cp424 ++import encodings.cp437 ++import encodings.cp500 ++import encodings.cp737 ++import encodings.cp775 ++import encodings.cp850 ++import encodings.cp852 ++import encodings.cp855 ++import encodings.cp856 ++import encodings.cp857 ++import encodings.cp860 ++import encodings.cp861 ++import encodings.cp862 ++import encodings.cp863 ++import encodings.cp864 ++import encodings.cp865 ++import encodings.cp866 ++import encodings.cp869 ++import encodings.cp874 ++import encodings.cp875 ++import encodings.cp932 ++import encodings.cp949 ++import encodings.cp950 ++import encodings.euc_jis_2004 ++import encodings.euc_jisx0213 ++import encodings.euc_jp ++import encodings.euc_kr ++import encodings.gb18030 ++import encodings.gb2312 ++import encodings.gbk ++import encodings.hex_codec ++import encodings.hp_roman8 ++import encodings.hz ++import encodings.idna ++import encodings.iso2022_jp_1 ++import encodings.iso2022_jp_2004 ++import encodings.iso2022_jp_2 ++import encodings.iso2022_jp_3 ++import encodings.iso2022_jp_ext ++import encodings.iso2022_jp ++import encodings.iso2022_kr ++import encodings.iso8859_10 ++import encodings.iso8859_11 ++import encodings.iso8859_13 ++import encodings.iso8859_14 ++import encodings.iso8859_15 ++import encodings.iso8859_16 ++import encodings.iso8859_1 ++import encodings.iso8859_2 ++import encodings.iso8859_3 ++import encodings.iso8859_4 ++import encodings.iso8859_5 ++import encodings.iso8859_6 ++import encodings.iso8859_7 ++import encodings.iso8859_8 ++import encodings.iso8859_9 ++import encodings.johab ++import encodings.koi8_r ++import encodings.koi8_u ++import encodings.latin_1 ++import encodings.mac_cyrillic ++import encodings.mac_greek ++import encodings.mac_iceland ++import encodings.mac_latin2 ++import encodings.mac_roman ++import encodings.mac_turkish ++import encodings.mbcs ++import encodings.palmos ++import encodings.ptcp154 ++import encodings.punycode ++import encodings.quopri_codec ++import encodings.raw_unicode_escape ++import encodings.rot_13 ++import encodings.shift_jis_2004 ++import encodings.shift_jis ++import encodings.shift_jisx0213 ++import encodings.string_escape ++import encodings.tis_620 ++import encodings.undefined ++import encodings.unicode_escape ++import encodings.unicode_internal ++import encodings.utf_16_be ++import encodings.utf_16_le ++import encodings.utf_16 ++import encodings.utf_7 ++import encodings.utf_8 ++import encodings.uu_codec ++import encodings.zlib_codec ++import cStringIO ++import gc ++import xml.dom ++except: ++pass ++ + from BitTorrent.platform import install_translation + install_translation() + Index: sbin/si_installbtimage === --- sbin/si_installbtimage (revision 0) +++ sbin/si_installbtimage (revision 0) @@ -0,0 +1,298 @@ +#!/usr/bin/perl -w +# +# "SystemImager" +# +# Copyright (C) 2005 Andrea Righi <[EMAIL PROTECTED]> + +use lib
Re: [Sisuite-devel] Re: Bit-Torrent transport
As discussed with Bernard and Brian on IRC I've moved the python bittorrent client into the BOEL initrd. It seems to work fine. I'll have to add the code to download boel_binaries, scripts and overrides via bittorrent. Moreover the python bittorrent client has a nice command (launchmany-console) that allows to run a seeder for more .torrent files. This could be very useful to run a single seeder on the image server (and maybe also in the clients) to distribute image, overrides, scripts and BOEL binaries at the same time... I'll try to integrate all this stuff ASAP and I'll synchronize the work with the Bernard's patch (regarding flamethrower integration). Regards, -Andrea Index: patches/bittorrent.freezehack.patch === --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++import re ++import encodings ++import encodings.ascii ++import encodings.base64_codec ++import encodings.big5hkscs ++import encodings.big5 ++import encodings.bz2_codec ++import encodings.charmap ++import encodings.cp037 ++import encodings.cp1006 ++import encodings.cp1026 ++import encodings.cp1140 ++import encodings.cp1250 ++import encodings.cp1251 ++import encodings.cp1252 ++import encodings.cp1253 ++import encodings.cp1254 ++import encodings.cp1255 ++import encodings.cp1256 ++import encodings.cp1257 ++import encodings.cp1258 ++import encodings.cp424 ++import encodings.cp437 ++import encodings.cp500 ++import encodings.cp737 ++import encodings.cp775 ++import encodings.cp850 ++import encodings.cp852 ++import encodings.cp855 ++import encodings.cp856 ++import encodings.cp857 ++import encodings.cp860 ++import encodings.cp861 ++import encodings.cp862 ++import encodings.cp863 ++import encodings.cp864 ++import encodings.cp865 ++import encodings.cp866 ++import encodings.cp869 ++import encodings.cp874 ++import encodings.cp875 ++import encodings.cp932 ++import encodings.cp949 ++import encodings.cp950 ++import encodings.euc_jis_2004 ++import encodings.euc_jisx0213 ++import encodings.euc_jp ++import encodings.euc_kr ++import encodings.gb18030 ++import encodings.gb2312 ++import encodings.gbk ++import encodings.hex_codec ++import encodings.hp_roman8 ++import encodings.hz ++import encodings.idna ++import encodings.iso2022_jp_1 ++import encodings.iso2022_jp_2004 ++import encodings.iso2022_jp_2 ++import encodings.iso2022_jp_3 ++import encodings.iso2022_jp_ext ++import encodings.iso2022_jp ++import encodings.iso2022_kr ++import encodings.iso8859_10 ++import encodings.iso8859_11 ++import encodings.iso8859_13 ++import encodings.iso8859_14 ++import encodings.iso8859_15 ++import encodings.iso8859_16 ++import encodings.iso8859_1 ++import encodings.iso8859_2 ++import encodings.iso8859_3 ++import encodings.iso8859_4 ++import encodings.iso8859_5 ++import encodings.iso8859_6 ++import encodings.iso8859_7 ++import encodings.iso8859_8 ++import encodings.iso8859_9 ++import encodings.johab ++import encodings.koi8_r ++import encodings.koi8_u ++import encodings.latin_1 ++import encodings.mac_cyrillic ++import encodings.mac_greek ++import encodings.mac_iceland ++import encodings.mac_latin2 ++import encodings.mac_roman ++import encodings.mac_turkish ++import encodings.mbcs ++import encodings.palmos ++import encodings.ptcp154 ++import encodings.punycode ++import encodings.quopri_codec ++import encodings.raw_unicode_escape ++import encodings.rot_13 ++import encodings.shift_jis_2004 ++import encodings.shift_jis ++import encodings.shift_jisx0213 ++import encodings.string_escape ++import encodings.tis_620 ++import encodings.undefined ++import encodings.unicode_escape ++import encodings.unicode_internal ++import encodings.utf_16_be ++import encodings.utf_16_le ++import encodings.utf_16 ++import encodings.utf_7 ++import encodings.utf_8 ++import encodings.uu_codec ++import encodings.zlib_codec ++import cStringIO ++import gc ++import xml.dom ++except: ++pass ++ + from BitTorrent.platform import install_translation + install_translation() + Index: sbin/si_installbtimage === --- sbin/si_installbtimage (revision 0) +++ sbin/si_installbtimage (revision 0) @@ -0,0 +1,290 @@ +#!/usr/bin/perl -w +# +# "SystemImager" +# +# Copyright (C) 2005
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi all, with this last pach I'd consider my bittorrent work ready to be checked into the trunk. Bernard, let me know what's your point, if you want to merge my changes in your patch, if you have comments, suggestions, etc. Otherwise I will proceed with the commit... Keep in mind that we can always rollback to the previous trunk and I can always switch to another bittorrent client without problem (I hope I've been quite modular..) Here is a short summary about the updates in the patch (I've also started to document something, so maybe I can send you ASAP a better detailed documentation...). With this patch I've added the command si_installbtimage and the init script /etc/init.d/systemimager-server/bittorrent. The first one is the command that provides to create the tarball of an image, the .torrent file and to launch the first seeder on the image server (see `perldoc si_installbtimage` for more info). To test the patch you must: 1) apply the patch to the trunk ;-) 2) re-run si_mkautoinstallscript for the image you want to install 3) define BITTORRENT_STAGING= in the kernel boot options (where is a directory in BOEL, so for example /tmp will use tmpfs and /a/tmp will use the /tmp space in the client's disk). 4) run the following commands no the image server: # /etc/init.d/systemimager-bittorrent start # si_installbtimage --image_server --image (add --compress if you want to create a tar.gz image file, instead of a plain tarball...) 5) boot the clients and wait for the installation... Best regards, -Andrea Index: patches/bittorrent.freezehack.patch === --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++import re ++import encodings ++import encodings.ascii ++import encodings.base64_codec ++import encodings.big5hkscs ++import encodings.big5 ++import encodings.bz2_codec ++import encodings.charmap ++import encodings.cp037 ++import encodings.cp1006 ++import encodings.cp1026 ++import encodings.cp1140 ++import encodings.cp1250 ++import encodings.cp1251 ++import encodings.cp1252 ++import encodings.cp1253 ++import encodings.cp1254 ++import encodings.cp1255 ++import encodings.cp1256 ++import encodings.cp1257 ++import encodings.cp1258 ++import encodings.cp424 ++import encodings.cp437 ++import encodings.cp500 ++import encodings.cp737 ++import encodings.cp775 ++import encodings.cp850 ++import encodings.cp852 ++import encodings.cp855 ++import encodings.cp856 ++import encodings.cp857 ++import encodings.cp860 ++import encodings.cp861 ++import encodings.cp862 ++import encodings.cp863 ++import encodings.cp864 ++import encodings.cp865 ++import encodings.cp866 ++import encodings.cp869 ++import encodings.cp874 ++import encodings.cp875 ++import encodings.cp932 ++import encodings.cp949 ++import encodings.cp950 ++import encodings.euc_jis_2004 ++import encodings.euc_jisx0213 ++import encodings.euc_jp ++import encodings.euc_kr ++import encodings.gb18030 ++import encodings.gb2312 ++import encodings.gbk ++import encodings.hex_codec ++import encodings.hp_roman8 ++import encodings.hz ++import encodings.idna ++import encodings.iso2022_jp_1 ++import encodings.iso2022_jp_2004 ++import encodings.iso2022_jp_2 ++import encodings.iso2022_jp_3 ++import encodings.iso2022_jp_ext ++import encodings.iso2022_jp ++import encodings.iso2022_kr ++import encodings.iso8859_10 ++import encodings.iso8859_11 ++import encodings.iso8859_13 ++import encodings.iso8859_14 ++import encodings.iso8859_15 ++import encodings.iso8859_16 ++import encodings.iso8859_1 ++import encodings.iso8859_2 ++import encodings.iso8859_3 ++import encodings.iso8859_4 ++import encodings.iso8859_5 ++import encodings.iso8859_6 ++import encodings.iso8859_7 ++import encodings.iso8859_8 ++import encodings.iso8859_9 ++import encodings.johab ++import encodings.koi8_r ++import encodings.koi8_u ++import encodings.latin_1 ++import encodings.mac_cyrillic ++import encodings.mac_greek ++import encodings.mac_iceland ++import encodings.mac_latin2 ++import encodings.mac_roman ++import encodings.mac_turkish ++import encodings.mbcs ++import encodings.palmos ++import encodings.ptcp154 ++import encodings.punycode ++import encodings.quopri_codec ++import encodings.raw_unicode_escape ++import encodings.rot_13 ++import encodings.shift_jis_2004 ++import encodings.shift_jis ++import
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi, here are some thoughts about updating (with a test install on a 2.4GHz Xeon with RHEL4 in mind): - the full image directory requires 711MB - the tarred up image needs: 640MB - built from the image directory in 31s - compressed image tarfile: 219MB - compress time (gzip) 100s - compress time (gzip -9) 270s - uncompress time 15s For the initial deployment it fully makes sense to work with the gzipped tarfile, this saves a lot of bandwidth and certainly speeds up things even on small clusters. For updates it seems to be a big waste to use the full tar. A tar containing only the differences between the image directory and the initial deployment tarfile would be much faster to deploy. An idea is to use two tars right from the start: image.tar.gz : full archive of the image directory (initial deployment image) image-diff.tar.gz : archive containing the diffs between the image directory and image.tar.gz . We would use only one of these differential archives (not incremental, just base+diff). At the initial deployment time image-diff.tar.gz would be an empty archive. The attached script "tardiff" builds such differential tars. It mainly spends its time with comparing the original image tarfile with the directory content on the filesystem. Actually most time is spent with decompressing the archive. Here are some numbers (with only small differences between image directory and image.tar.gz): - if the tarfile is compressed (image.tar.gz) building image-diff.tar.gz takes 17.6s - if the tarfile is not compressed (image.tar), building image-diff.tar.gz takes 3.8s. So it is really feasible to build (and overwrite) image-diff.tar.gz right before upgrading a cluster. The benefit of using this instead of a full tar is a very much reduced amount of data to be transfered over the network and the fact that unpacking this small archive really touches only the modified files, nothing else. Any thoughts? If this makes sense, I'll try integrating once there is some BT transport available (or the framework around it). Regards, Erich On Monday 02 January 2006 23:42, Paul Greidanus wrote: > Hi Andrea and Bernard, > > I'm just wondering if there is (or is planned) to have a way to use BT > to run si_updateclient? or is this for initial deployment only? I've > always thought systemimager's strongest feature is the update client > feature, but I've had trouble if I tell a large number (>50) machines to > update at the same time.. it would be cool if BT would fix this one, and > just have the nodes sync to eachother.. > > Andrea Righi wrote: > > Hi all, > > > > attached is some work I've done regarding bittorrent... > > > > Since this seems to work and since it can be useful also for those that > > are working for the bittorrent porting (in particular for Bernard) I've > > decided to post this patch (based on the developer trunk). > > > > I've used the standard bt client (written in python) both for clients > > and image server, so I've included both the python interpreter and > > bittorrent scripts in BOEL binaries. > > > > Bernard is using libBt, it seems more light since it doesn't require > > python, maybe it could be interesting do some tests to evaluate the > > difference in performances between the two products. > > > > To use my bittorrent patch you've to start the tracker and the first > > seeder on the image server, simply using the following command: > > > > # si_startbttrack --image_server --image > > > > This command creates also a single tar file for the image specified by > > comdline; run with --update_image to update the tar file. > > > > I've not written the si_stopbttrack command, even if it is very > > simple... but for now you have to stop it only killing the tracker and > > the seeder manually... ;-) > > > > For the client side I don't use flamethrower, the bt client works alone > > without further packages (except python, of course). > > > > You must only define BITTORRENT_STAGING= as a kernel boot > > parameter. In this way the bittorrent protocol is used to deploy the > > image in the clients. First of all the image is downloaded in a staging > > directory (the path is specified by the parameter), then it is untarred > > in /a/ (the root directory of the installed fs) and the tar is removed > > when the exctraction is completed. > > > > If the client has not a lot of memory, like my test machine :-(, you can > > deploy the tar file using the client disk space, for example I usually > > use BITTORRENT_STAGE=/a/tmp... otherwise if you have enough RAM, to > > increase performance, is better to deploy the tar file in tmpfs (for > > example using /tmp - we can assume this as the default option). > > > > If you can do some tests let me know your feedback... in particular for > > Bernard that it's working hardly with
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi Bernard, I think it is very fortunate that your and Andrea's approach are different and that the implementations are done in separate parts of the code. This way we will perhaps be able to test both and compile the optimal solution out of both. In the kernel world it is quite normal to have two components doing the same job in a slightly different way. And there's often somebody morphing an even better solution out of the two ones. Can you submit your changes as patch? I remember the flamethrower to need a lot of preparations like - setting up the flamethrower.conf - defining the "modules" - allocating unique ports for every module - re-starting flamethrowerd (the senders) - getting the ports info to the nodes - ... Some of the steps seem similar to bittorrent but some can certainly be skipped. I'd be curious to see how you deal with this in your approach. But actually I am thinking about Paul's question and how to deal with updating an image in bittorrent way. To play with that I need _some_ implementation of the BT transport. Regards, Erich On Monday 09 January 2006 20:07, Bernard Li wrote: > I'm still waiting on Brian to comment on whether to integrate it as a > flamethrower transport or separately (like how Andrea has done it). > > I think my code already have some integration with Andrea's work - the > major difference now is he is using the official python BT and I'm using > libBT. > > I should be able to check my stuff in soon. > > Cheers, > > Bernard --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click ___ Sisuite-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sisuite-devel
RE: [Sisuite-devel] Re: Bit-Torrent transport
I'm still waiting on Brian to comment on whether to integrate it as a flamethrower transport or separately (like how Andrea has done it). I think my code already have some integration with Andrea's work - the major difference now is he is using the official python BT and I'm using libBT. I should be able to check my stuff in soon. Cheers, Bernard > -Original Message- > From: Andrea Righi [mailto:[EMAIL PROTECTED] > Sent: Monday, January 09, 2006 10:53 > To: [EMAIL PROTECTED] > Cc: Bernard Li; Brian Elliott Finley; [EMAIL PROTECTED]; SIS Devel > Subject: Re: [Sisuite-devel] Re: Bit-Torrent transport > > Erich Focht wrote: > > Hi Andrea, > > > > do you (or anybody else) mind checking this into trunk? > It's easier to play > > with it from there and it looks already quite usable. > > Yes, we could check... maybe we've to synchronize with > Bernard to merge > both our work and we've to choose if we want to use libbt or the > standard client, but this can be independent from checking > something in > the trunk... we can always rollback to a previous version, or > change the > client afterwards. Moreover playing with svn is easier than applying > some patches.. :) > > Regards, > -Andrea > > > > > Regards, > > Erich > > > --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_idv37&alloc_id865&op=click ___ Sisuite-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sisuite-devel
Re: [Sisuite-devel] Re: Bit-Torrent transport
Erich Focht wrote: Hi Andrea, do you (or anybody else) mind checking this into trunk? It's easier to play with it from there and it looks already quite usable. Yes, we could check... maybe we've to synchronize with Bernard to merge both our work and we've to choose if we want to use libbt or the standard client, but this can be independent from checking something in the trunk... we can always rollback to a previous version, or change the client afterwards. Moreover playing with svn is easier than applying some patches.. :) Regards, -Andrea Regards, Erich --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click ___ Sisuite-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sisuite-devel
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi Andrea, do you (or anybody else) mind checking this into trunk? It's easier to play with it from there and it looks already quite usable. Regards, Erich On Monday 09 January 2006 10:08, Andrea Righi wrote: > Hi all, > > other work for the bittorrent transport... > > In this patch I've added to BOEL binaries the standard bittorrent python > client "freezed" as a normal ELF binary, as suggested by Bernard in a > previous chat with him. > > This is a very important issue, because with the python interpreter the > BOEL binaries took a lot of memory in the clients and a lot of time to > be deployed... This is because I've always prefered the Bernard approach > using libBT... :), but now also with this approach the size of the > binaries is strongly reduced (from 20-30MB to 6.7MB!). > > To create the bittorrent binaries from the python scripts, I've used > cx_Freeze (http://starship.python.net/crew/atuining/cx_Freeze/). > > Moreover I've implemented a new heuristic for a client to decide when to > stop seeding and go forward with the installation. In particular parsing > the bittorrent-console output is possible to get the upload rate (= how > many KB/s are pushed to the other peers). Looking periodically at this > value is possible to decide that under a certain minimum threshold the > client/seeder is no more useful, and it can proceed to the final steps > of the installation (untar the image, run systemconfigurator, etc.). > > I've done some test and it seems to work very nice! Unfortunately at the > moment I've only 1 real client for the tests... so I had to do some > tricks to simulate more clients on another installed machine running > more bittorrent-console on it... :-( this is very bad, anyway I suppose > it should be a simple but quite reliable approximation... so who as 100 > clients to do some better tests? :-)) > > Comments are welcome... > > Best regards, > -Andrea --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click ___ Sisuite-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sisuite-devel
RE: [Sisuite-devel] Re: Bit-Torrent transport
Title: Re: [Sisuite-devel] Re: Bit-Torrent transport Hey Andrea: What speed did you get transferring between one peer to another peer? As I have mentioned to you with my tests, I was only able to get 1.5MBytes/sec max (with --max_upload_rate 0 on a 100MBit network). With LibBT's btget, there is the option (-q 1) to quit when all peers have finished, this seems to be a good functionality to use. Did your bittorrent frozen binaries require the Python libraries to run? Cheers, Bernard From: Andrea Righi [mailto:[EMAIL PROTECTED]Sent: Mon 09/01/2006 01:08To: [EMAIL PROTECTED]Cc: Bernard Li; Brian Elliott Finley; [EMAIL PROTECTED]; [EMAIL PROTECTED]; SIS DevelSubject: Re: [Sisuite-devel] Re: Bit-Torrent transport Hi all,other work for the bittorrent transport...In this patch I've added to BOEL binaries the standard bittorrent pythonclient "freezed" as a normal ELF binary, as suggested by Bernard in aprevious chat with him.This is a very important issue, because with the python interpreter theBOEL binaries took a lot of memory in the clients and a lot of time tobe deployed... This is because I've always prefered the Bernard approachusing libBT... :), but now also with this approach the size of thebinaries is strongly reduced (from 20-30MB to 6.7MB!).To create the bittorrent binaries from the python scripts, I've usedcx_Freeze (http://starship.python.net/crew/atuining/cx_Freeze/).Moreover I've implemented a new heuristic for a client to decide when tostop seeding and go forward with the installation. In particular parsingthe bittorrent-console output is possible to get the upload rate (= howmany KB/s are pushed to the other peers). Looking periodically at thisvalue is possible to decide that under a certain minimum threshold theclient/seeder is no more useful, and it can proceed to the final stepsof the installation (untar the image, run systemconfigurator, etc.).I've done some test and it seems to work very nice! Unfortunately at themoment I've only 1 real client for the tests... so I had to do sometricks to simulate more clients on another installed machine runningmore bittorrent-console on it... :-( this is very bad, anyway I supposeit should be a simple but quite reliable approximation... so who as 100clients to do some better tests? :-))Comments are welcome...Best regards,-Andrea
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi all, other work for the bittorrent transport... In this patch I've added to BOEL binaries the standard bittorrent python client "freezed" as a normal ELF binary, as suggested by Bernard in a previous chat with him. This is a very important issue, because with the python interpreter the BOEL binaries took a lot of memory in the clients and a lot of time to be deployed... This is because I've always prefered the Bernard approach using libBT... :), but now also with this approach the size of the binaries is strongly reduced (from 20-30MB to 6.7MB!). To create the bittorrent binaries from the python scripts, I've used cx_Freeze (http://starship.python.net/crew/atuining/cx_Freeze/). Moreover I've implemented a new heuristic for a client to decide when to stop seeding and go forward with the installation. In particular parsing the bittorrent-console output is possible to get the upload rate (= how many KB/s are pushed to the other peers). Looking periodically at this value is possible to decide that under a certain minimum threshold the client/seeder is no more useful, and it can proceed to the final steps of the installation (untar the image, run systemconfigurator, etc.). I've done some test and it seems to work very nice! Unfortunately at the moment I've only 1 real client for the tests... so I had to do some tricks to simulate more clients on another installed machine running more bittorrent-console on it... :-( this is very bad, anyway I suppose it should be a simple but quite reliable approximation... so who as 100 clients to do some better tests? :-)) Comments are welcome... Best regards, -Andrea Index: patches/bittorrent.freezehack.patch === --- patches/bittorrent.freezehack.patch (revision 0) +++ patches/bittorrent.freezehack.patch (revision 0) @@ -0,0 +1,125 @@ +--- bittorrent-console.py.orig 2006-01-08 18:44:54.374123448 +0100 bittorrent-console.py 2006-01-08 18:46:56.425568808 +0100 +@@ -14,6 +14,122 @@ + + from __future__ import division + ++try: ++import re ++import encodings ++import encodings.ascii ++import encodings.base64_codec ++import encodings.big5hkscs ++import encodings.big5 ++import encodings.bz2_codec ++import encodings.charmap ++import encodings.cp037 ++import encodings.cp1006 ++import encodings.cp1026 ++import encodings.cp1140 ++import encodings.cp1250 ++import encodings.cp1251 ++import encodings.cp1252 ++import encodings.cp1253 ++import encodings.cp1254 ++import encodings.cp1255 ++import encodings.cp1256 ++import encodings.cp1257 ++import encodings.cp1258 ++import encodings.cp424 ++import encodings.cp437 ++import encodings.cp500 ++import encodings.cp737 ++import encodings.cp775 ++import encodings.cp850 ++import encodings.cp852 ++import encodings.cp855 ++import encodings.cp856 ++import encodings.cp857 ++import encodings.cp860 ++import encodings.cp861 ++import encodings.cp862 ++import encodings.cp863 ++import encodings.cp864 ++import encodings.cp865 ++import encodings.cp866 ++import encodings.cp869 ++import encodings.cp874 ++import encodings.cp875 ++import encodings.cp932 ++import encodings.cp949 ++import encodings.cp950 ++import encodings.euc_jis_2004 ++import encodings.euc_jisx0213 ++import encodings.euc_jp ++import encodings.euc_kr ++import encodings.gb18030 ++import encodings.gb2312 ++import encodings.gbk ++import encodings.hex_codec ++import encodings.hp_roman8 ++import encodings.hz ++import encodings.idna ++import encodings.iso2022_jp_1 ++import encodings.iso2022_jp_2004 ++import encodings.iso2022_jp_2 ++import encodings.iso2022_jp_3 ++import encodings.iso2022_jp_ext ++import encodings.iso2022_jp ++import encodings.iso2022_kr ++import encodings.iso8859_10 ++import encodings.iso8859_11 ++import encodings.iso8859_13 ++import encodings.iso8859_14 ++import encodings.iso8859_15 ++import encodings.iso8859_16 ++import encodings.iso8859_1 ++import encodings.iso8859_2 ++import encodings.iso8859_3 ++import encodings.iso8859_4 ++import encodings.iso8859_5 ++import encodings.iso8859_6 ++import encodings.iso8859_7 ++import encodings.iso8859_8 ++import encodings.iso8859_9 ++import encodings.johab ++import encodings.koi8_r ++import encodings.koi8_u ++import encodings.latin_1 ++import encodings.mac_cyrillic ++import encodings.mac_greek ++import encodings.mac_iceland ++import encodings.mac_latin2 ++import encodings.mac_roman ++import encodings.mac_turkish ++import encodings.mbcs ++import encodings.palmos ++import encodings.ptcp154 ++import encodings.punycode ++import encodings.quopri_codec ++import encodings.raw_unicode_escape
RE: [Sisuite-devel] Re: Bit-Torrent transport
Title: Re: [Sisuite-devel] Re: Bit-Torrent transport
Steve mentioned using "ar" in
place of tar so that we can update the archive - anybody have experience with
this?
Cheers,
Bernard
From: [EMAIL PROTECTED]
on behalf of Bernard LiSent: Mon 02/01/2006 15:46To: Paul
Greidanus; [EMAIL PROTECTED]Cc: [EMAIL PROTECTED]; SIS
DevelSubject: RE: [Sisuite-devel] Re: Bit-Torrent
transport
Well this would be tricky
because right now we are creating a tarball of the image directory, as far as
"update-clients" is concerned we are re-deploying as there are no previous
information about the image. (you need to re-create the tarball and torrent file
each time)
If we can somehow modify how
bittorrent works making it possible to BT the entire directory (skipping special
directories like /dev), then this may be possible. Though I'm not sure how
the BT protocol treats file updates... i.e. I created a torrent of a
directory, seed it, then I change some files, would BT be able to handle the
changes?
Cheers,
Bernard
From: Paul Greidanus
[mailto:[EMAIL PROTECTED]Sent: Mon 02/01/2006
14:42To: [EMAIL PROTECTED]Cc: Bernard Li;
[EMAIL PROTECTED]; SIS DevelSubject: Re: [Sisuite-devel] Re:
Bit-Torrent transport
Hi Andrea and Bernard,I'm just wondering if there is (or
is planned) to have a way to use BTto run si_updateclient? or is this for
initial deployment only? I'vealways thought systemimager's strongest feature
is the update clientfeature, but I've had trouble if I tell a large number
(>50) machines toupdate at the same time.. it would be cool if BT would
fix this one, andjust have the nodes sync to eachother..Andrea Righi
wrote:> Hi all,>> attached is some work I've done regarding
bittorrent...>> Since this seems to work and since it can be
useful also for those that> are working for the bittorrent porting (in
particular for Bernard) I've> decided to post this patch (based on the
developer trunk).>> I've used the standard bt client (written in
python) both for clients> and image server, so I've included both the
python interpreter and> bittorrent scripts in BOEL
binaries.>> Bernard is using libBt, it seems more light since it
doesn't require> python, maybe it could be interesting do some tests to
evaluate the> difference in performances between the two
products.>> To use my bittorrent patch you've to start the tracker
and the first> seeder on the image server, simply using the following
command:>> # si_startbttrack --image_server
--image >> This command
creates also a single tar file for the image specified by> comdline; run
with --update_image to update the tar file.>> I've not written the
si_stopbttrack command, even if it is very> simple... but for now you
have to stop it only killing the tracker and> the seeder manually...
;-)>> For the client side I don't use flamethrower, the bt client
works alone> without further packages (except python, of
course).>> You must only define
BITTORRENT_STAGING= as a kernel boot> parameter. In this
way the bittorrent protocol is used to deploy the> image in the clients.
First of all the image is downloaded in a staging> directory (the path is
specified by the parameter), then it is untarred> in /a/ (the root
directory of the installed fs) and the tar is removed> when the
exctraction is completed.>> If the client has not a lot of memory,
like my test machine :-(, you can> deploy the tar file using the client
disk space, for example I usually> use BITTORRENT_STAGE=/a/tmp...
otherwise if you have enough RAM, to> increase performance, is better to
deploy the tar file in tmpfs (for> example using /tmp - we can assume
this as the default option).>> If you can do some tests let me
know your feedback... in particular for> Bernard that it's working hardly
with bittorrent... ;-)>> Best regards and happy new year!>
-Andrea>>>
>>
diff -urN trunk/etc/autoinstallscript.template
bittorrent/etc/autoinstallscript.template> ---
trunk/etc/autoinstallscript.template 2006-01-02
09:40:53.671461104 +0100> +++ bittorrent/etc/autoinstallscript.template
2006-01-02 09:44:25.384275864 +0100> @@ -216,6 +216,29
@@> if [ ! -z $MONITOR_SERVER ];
then>
stop_report_task 100> fi> +elif [ !
-z $BITTORRENT_STAGING ]; then> +
bittorrent_log=/tmp/bittorrent.log> +
bittorrent_polling_time=30> + logmsg "Start downloading
image using torrent ${IMAGENAME}.tar.torrent"> + cd
${BITTORRENT_STAGING} && /usr/bin/python /usr/bin/bittorrent-console
--no_start_trackerless_client --max_upload_rate> +
pid=$!> +
Re: [Sisuite-devel] Re: Bit-Torrent transport
A new patch that fixes some monitoring errors using bittorrent...
Regards,
-Andrea
diff -urN trunk/etc/autoinstallscript.template bittorrent/etc/autoinstallscript.template
--- trunk/etc/autoinstallscript.template 2006-01-02 09:40:53.0 +0100
+++ bittorrent/etc/autoinstallscript.template 2006-01-02 22:24:05.0 +0100
@@ -216,6 +216,29 @@
if [ ! -z $MONITOR_SERVER ]; then
stop_report_task 100
fi
+elif [ ! -z $BITTORRENT_STAGING ]; then
+bittorrent_log=/tmp/bittorrent.log
+bittorrent_polling_time=30
+logmsg "Start downloading image using torrent ${IMAGENAME}.tar.torrent"
+cd ${BITTORRENT_STAGING} && /usr/bin/python /usr/bin/bittorrent-console --no_start_trackerless_client --max_upload_rate 0 --display_interval 1 --bind ${IPADDR} /scripts/${IMAGENAME}.tar.torrent > $bittorrent_log &
+pid=$!
+while :; do
+status=`grep "percent done:" $bittorrent_log | sed -ne '$p' | sed 's/percent done: *//'`
+[ -z "$status" ] && status=0.0
+echo "percent done: $status %"
+if [ "$status" = "100.0" ]; then
+kill -9 $pid
+unset bittorrent_log pid
+break
+fi
+sleep $bittorrent_polling_time
+done
+unset bittorrent_polling_time
+if [ ! -z $MONITOR_SERVER ]; then
+stop_report_task 100
+fi
+logmsg "Extracting image ${IMAGENAME}.tar ..."
+cd /a/ && tar -xvf ${BITTORRENT_STAGING}/${IMAGENAME}.tar && rm -f ${BITTORRENT_STAGING}/${IMAGENAME}.tar
else
# Use rsync
if [ $NO_LISTING ]; then
diff -urN trunk/initrd_source/skel/etc/init.d/functions bittorrent/initrd_source/skel/etc/init.d/functions
--- trunk/initrd_source/skel/etc/init.d/functions 2006-01-02 23:30:45.0 +0100
+++ bittorrent/initrd_source/skel/etc/init.d/functions 2006-01-02 23:33:48.0 +0100
@@ -193,6 +193,8 @@
echo "MONITOR_SERVER=$MONITOR_SERVER" >> /tmp/variables.txt
echo "MONITOR_PORT=$MONITOR_PORT" >> /tmp/variables.txt
echo "MONITOR_CONSOLE=$MONITOR_CONSOLE" >> /tmp/variables.txt
+
+echo "BITTORRENT_STAGING=$BITTORRENT_STAGING" >> /tmp/variables.txt
}
#
@@ -1418,7 +1420,11 @@
# Evaluate image size.
logmsg "Evaluating image size."
-IMAGESIZE=`rsync -av $IMAGESERVER::$IMAGENAME | grep "total size" | sed -e "s/total size is \([0-9]*\).*/\1/"`
+if [ -z $BITTORRENT_STAGING ]; then
+IMAGESIZE=`rsync -av $IMAGESERVER::$IMAGENAME | grep "total size" | sed -e "s/total size is \([0-9]*\).*/\1/"`
+else
+IMAGESIZE=`python /usr/bin/torrentinfo-console /scripts/$IMAGENAME.tar.torrent | sed -ne "s/file size\.*: \([0-9]*\) .*$/\1/p"`
+fi
IMAGESIZE=`expr $IMAGESIZE / 1024`
# Evaluate disks size.
diff -urN trunk/make.d/bittorrent.rul bittorrent/make.d/bittorrent.rul
--- trunk/make.d/bittorrent.rul 1970-01-01 01:00:00.0 +0100
+++ bittorrent/make.d/bittorrent.rul 2006-01-02 09:51:18.0 +0100
@@ -0,0 +1,31 @@
+#
+# 2005-12-28 Andrea Righi
+# - Bittorrent scripts
+#
+
+
+BITTORRENT_VERSION := 4.3.4
+BITTORRENT_DIR := BitTorrent-$(BITTORRENT_VERSION)
+BITTORRENT_TARBALL := BitTorrent-$(BITTORRENT_VERSION).tar.gz
+BITTORRENT_URL := http://www.bittorrent.com/dl/$(BITTORRENT_TARBALL)
+BITTORRENT_BINARY := $(SRC_DIR)/$(BITTORRENT_DIR)/bittorrent.py
+
+PHONY += bittorrent
+
+bittorrent: $(BITTORRENT_BINARY)
+
+$(BITTORRENT_BINARY): $(SRC_DIR)/$(BITTORRENT_TARBALL)
+ rm -rf $(SRC_DIR)/$(BITTORRENT_DIR)
+ cd $(SRC_DIR) && tar -xvzf $(BITTORRENT_TARBALL)
+
+$(SRC_DIR)/$(BITTORRENT_TARBALL):
+ [ -d $(SRC_DIR) ] || mkdir -p $(SRC_DIR)
+ $(GETSOURCE) $(BITTORRENT_URL) $(SRC_DIR)
+
+ALL_SOURCE += $(SRC_DIR)/$(BITTORRENT_TARBALL)
+
+PHONY += bittorrent_clean
+bittorrent_clean:
+ rm -rf $(SRC_DIR)/$(BITTORRENT_DIR)
+
+# /* vi: set noet ts=4: */
diff -urN trunk/make.d/python.rul bittorrent/make.d/python.rul
--- trunk/make.d/python.rul 1970-01-01 01:00:00.0 +0100
+++ bittorrent/make.d/python.rul 2006-01-02 09:48:55.0 +0100
@@ -0,0 +1,33 @@
+#
+# 2005-12-20 Andrea Righi
+# - Python interpreter
+#
+
+
+PYTHON_VERSION := 2.4.2
+PYTHON_DIR := Python-$(PYTHON_VERSION)
+PYTHON_TARBALL := Python-$(PYTHON_VERSION).tar.bz2
+PYTHON_URL := http://www.python.org/ftp/python/2.4.2/$(PYTHON_TARBALL)
+PYTHON_BINARY := $(SRC_DIR)/$(PYTHON_DIR)/python
+
+PHONY += python
+python: $(PYTHON_BINARY)
+$(PYTHON_BINARY): $(SRC_DIR)/$(PYTHON_TARBALL)
+ rm -rf $(SRC_DIR)/$(PYTHON_DIR)
+ cd $(SRC_DIR) && tar -xvjf $(PYTHON_TARBALL)
+ cd $(SRC_DIR)/$(PYTHON_DIR) && \
+./configure --prefix=/usr --disable-shared --disable-ipv6 --disable-profiling --without-tsc --without-pymalloc --disable-toolbox-glue CFLAGS="-Os"
+ export CFLAGS="-Os"
+ $(MAKE) -C $(SRC_DIR)/$(PYTHON_DIR) CFLAGS="-Os"
+
+$(SRC_DIR)/$(PYTHON_TARBALL):
+ [ -d $(SRC_DIR) ] || mkdir -p
RE: [Sisuite-devel] Re: Bit-Torrent transport
Title: Re: [Sisuite-devel] Re: Bit-Torrent transport
Well this would be tricky
because right now we are creating a tarball of the image directory, as far as
"update-clients" is concerned we are re-deploying as there are no previous
information about the image. (you need to re-create the tarball and torrent file
each time)
If we can somehow modify how
bittorrent works making it possible to BT the entire directory (skipping special
directories like /dev), then this may be possible. Though I'm not sure how
the BT protocol treats file updates... i.e. I created a torrent of a
directory, seed it, then I change some files, would BT be able to handle the
changes?
Cheers,
Bernard
From: Paul Greidanus
[mailto:[EMAIL PROTECTED]Sent: Mon 02/01/2006
14:42To: [EMAIL PROTECTED]Cc: Bernard Li;
[EMAIL PROTECTED]; SIS DevelSubject: Re: [Sisuite-devel] Re:
Bit-Torrent transport
Hi Andrea and Bernard,I'm just wondering if there is (or
is planned) to have a way to use BTto run si_updateclient? or is this for
initial deployment only? I'vealways thought systemimager's strongest feature
is the update clientfeature, but I've had trouble if I tell a large number
(>50) machines toupdate at the same time.. it would be cool if BT would
fix this one, andjust have the nodes sync to eachother..Andrea Righi
wrote:> Hi all,>> attached is some work I've done regarding
bittorrent...>> Since this seems to work and since it can be
useful also for those that> are working for the bittorrent porting (in
particular for Bernard) I've> decided to post this patch (based on the
developer trunk).>> I've used the standard bt client (written in
python) both for clients> and image server, so I've included both the
python interpreter and> bittorrent scripts in BOEL
binaries.>> Bernard is using libBt, it seems more light since it
doesn't require> python, maybe it could be interesting do some tests to
evaluate the> difference in performances between the two
products.>> To use my bittorrent patch you've to start the tracker
and the first> seeder on the image server, simply using the following
command:>> # si_startbttrack --image_server
--image >> This command
creates also a single tar file for the image specified by> comdline; run
with --update_image to update the tar file.>> I've not written the
si_stopbttrack command, even if it is very> simple... but for now you
have to stop it only killing the tracker and> the seeder manually...
;-)>> For the client side I don't use flamethrower, the bt client
works alone> without further packages (except python, of
course).>> You must only define
BITTORRENT_STAGING= as a kernel boot> parameter. In this
way the bittorrent protocol is used to deploy the> image in the clients.
First of all the image is downloaded in a staging> directory (the path is
specified by the parameter), then it is untarred> in /a/ (the root
directory of the installed fs) and the tar is removed> when the
exctraction is completed.>> If the client has not a lot of memory,
like my test machine :-(, you can> deploy the tar file using the client
disk space, for example I usually> use BITTORRENT_STAGE=/a/tmp...
otherwise if you have enough RAM, to> increase performance, is better to
deploy the tar file in tmpfs (for> example using /tmp - we can assume
this as the default option).>> If you can do some tests let me
know your feedback... in particular for> Bernard that it's working hardly
with bittorrent... ;-)>> Best regards and happy new year!>
-Andrea>>>
>>
diff -urN trunk/etc/autoinstallscript.template
bittorrent/etc/autoinstallscript.template> ---
trunk/etc/autoinstallscript.template 2006-01-02
09:40:53.671461104 +0100> +++ bittorrent/etc/autoinstallscript.template
2006-01-02 09:44:25.384275864 +0100> @@ -216,6 +216,29
@@> if [ ! -z $MONITOR_SERVER ];
then>
stop_report_task 100> fi> +elif [ !
-z $BITTORRENT_STAGING ]; then> +
bittorrent_log=/tmp/bittorrent.log> +
bittorrent_polling_time=30> + logmsg "Start downloading
image using torrent ${IMAGENAME}.tar.torrent"> + cd
${BITTORRENT_STAGING} && /usr/bin/python /usr/bin/bittorrent-console
--no_start_trackerless_client --max_upload_rate> +
pid=$!> + while :; do>
+ status=`grep "percent done:"
$bittorrent_log | sed -ne '$p' | sed 's/percent done: *//'`>
+ [ -z "$status" ] &&
status=0.0> + echo "percent
done: $status %"> + if [
"$status" = "100.0" ]; then>
+ kill -9
$pid>
Re: [Sisuite-devel] Re: Bit-Torrent transport
Hi Andrea and Bernard,
I'm just wondering if there is (or is planned) to have a way to use BT
to run si_updateclient? or is this for initial deployment only? I've
always thought systemimager's strongest feature is the update client
feature, but I've had trouble if I tell a large number (>50) machines to
update at the same time.. it would be cool if BT would fix this one, and
just have the nodes sync to eachother..
Andrea Righi wrote:
Hi all,
attached is some work I've done regarding bittorrent...
Since this seems to work and since it can be useful also for those that
are working for the bittorrent porting (in particular for Bernard) I've
decided to post this patch (based on the developer trunk).
I've used the standard bt client (written in python) both for clients
and image server, so I've included both the python interpreter and
bittorrent scripts in BOEL binaries.
Bernard is using libBt, it seems more light since it doesn't require
python, maybe it could be interesting do some tests to evaluate the
difference in performances between the two products.
To use my bittorrent patch you've to start the tracker and the first
seeder on the image server, simply using the following command:
# si_startbttrack --image_server --image
This command creates also a single tar file for the image specified by
comdline; run with --update_image to update the tar file.
I've not written the si_stopbttrack command, even if it is very
simple... but for now you have to stop it only killing the tracker and
the seeder manually... ;-)
For the client side I don't use flamethrower, the bt client works alone
without further packages (except python, of course).
You must only define BITTORRENT_STAGING= as a kernel boot
parameter. In this way the bittorrent protocol is used to deploy the
image in the clients. First of all the image is downloaded in a staging
directory (the path is specified by the parameter), then it is untarred
in /a/ (the root directory of the installed fs) and the tar is removed
when the exctraction is completed.
If the client has not a lot of memory, like my test machine :-(, you can
deploy the tar file using the client disk space, for example I usually
use BITTORRENT_STAGE=/a/tmp... otherwise if you have enough RAM, to
increase performance, is better to deploy the tar file in tmpfs (for
example using /tmp - we can assume this as the default option).
If you can do some tests let me know your feedback... in particular for
Bernard that it's working hardly with bittorrent... ;-)
Best regards and happy new year!
-Andrea
diff -urN trunk/etc/autoinstallscript.template
bittorrent/etc/autoinstallscript.template
--- trunk/etc/autoinstallscript.template2006-01-02 09:40:53.671461104
+0100
+++ bittorrent/etc/autoinstallscript.template 2006-01-02 09:44:25.384275864
+0100
@@ -216,6 +216,29 @@
if [ ! -z $MONITOR_SERVER ]; then
stop_report_task 100
fi
+elif [ ! -z $BITTORRENT_STAGING ]; then
+bittorrent_log=/tmp/bittorrent.log
+bittorrent_polling_time=30
+logmsg "Start downloading image using torrent ${IMAGENAME}.tar.torrent"
+cd ${BITTORRENT_STAGING} && /usr/bin/python /usr/bin/bittorrent-console
--no_start_trackerless_client --max_upload_rate
+pid=$!
+while :; do
+status=`grep "percent done:" $bittorrent_log | sed -ne '$p' | sed
's/percent done: *//'`
+[ -z "$status" ] && status=0.0
+echo "percent done: $status %"
+if [ "$status" = "100.0" ]; then
+kill -9 $pid
+unset bittorrent_log pid
+break
+fi
+sleep $bittorrent_polling_time
+done
+unset bittorrent_polling_time
+if [ ! -z $MONITOR_SERVER ]; then
+stop_report_task 100
+fi
+logmsg "Extracting image ${IMAGENAME}.tar ..."
+cd /a/ && tar -xvf ${BITTORRENT_STAGING}/${IMAGENAME}.tar && rm -f
${BITTORRENT_STAGING}/${IMAGENAME}.tar
else
# Use rsync
if [ $NO_LISTING ]; then
diff -urN trunk/initrd_source/skel/etc/init.d/functions
bittorrent/initrd_source/skel/etc/init.d/functions
--- trunk/initrd_source/skel/etc/init.d/functions 2006-01-02
09:40:53.388504120 +0100
+++ bittorrent/initrd_source/skel/etc/init.d/functions 2006-01-02
09:44:40.177027024 +0100
@@ -193,6 +193,8 @@
echo "MONITOR_SERVER=$MONITOR_SERVER">> /tmp/variables.txt
echo "MONITOR_PORT=$MONITOR_PORT">> /tmp/variables.txt
echo "MONITOR_CONSOLE=$MONITOR_CONSOLE" >> /tmp/variables.txt
+
+echo "BITTORRENT_STAGING=$BITTORRENT_STAGING">>
/tmp/variables.txt
}
#
diff -urN trunk/make.d/bittorrent.rul bittorrent/make.d/bittorrent.rul
--- trunk/make.d/bittorrent.rul 1970-01-01 01:00:00.0 +0100
+++ bittorrent/make.d/bittorrent.rul2006-01-02
