Thus spake Bernard Li ([EMAIL PROTECTED]):
Hi Andrea:
I think it is possible to compile python code such that you can execute it
without the python interpreter - if you want, you can probably take a look into
that route, however it seems that libBT is working fine. In fact I wouldn't
mind inviting the developers of libBT to participate in our little project as
they may have better insight on how we can integrate their code into
SystemImager and may be interested in contributing code. But perhaps we need
to formalize our approach first before we contact them.
Concerning the branch - I was thinking of working straight off the trunk (if this is something which Brian wants).
Working off trunk is preferable, unless the work will cause
breakage to other bits of the system during development (such as with my
udev stuff).
I think as long as we are cautious in checking code in (making sure that code
checked in won't break the trunk), then it should be fine. Though I am not
opposed to the branching idea but we already have Brian's udev branch so the
more branches we have, the more time it takes to merge things back in. Just my
$0.02 :-)
Cheers,
Bernard
________________________________
From: Andrea Righi [mailto:[EMAIL PROTECTED]
Sent: Tue 27/12/2005 15:29
To: [EMAIL PROTECTED]
Cc: Bernard Li; Brian Elliott Finley; [EMAIL PROTECTED]
Subject: Re: Bit-Torrent transport
Very good about the tests made by Bernard!!
I've done other little tests using the standard client from
http://www.bittorrent.com/. So, first of all I've include python
compiled into the initrd without problem, but I've some troubles
including the bittorrent scripts... to put them I've just converted the
files from the rpm into a tarball and distributed as a
bittorrent_binaries.tar.gz rsynced at installation time... I'd want to
spend some more time in this, trying to do a cleaner installation, but
anyway if the LibBT is stable enough we can use it without problems.
Regarding the image distribution, I like the tarball approach, since it
is very quick, clean and simple. And we could resolve the memory problem
allowing to stage the image also in the client's disks; for example we
could specify a kernel boot paramenters, like BITTORRENT_STAGE=<dir>.
The default value will be /tmp (tmpfs ramdisk), but it could be also
/a/tmp, /a/scratch or /a/space_created_just_for_the_img_staging... after
the staging we have to simply extract the tarball in /a/ and remove it
at the end. In this way we can image also clients with a little ram size.
Moreover it could be interesting to have a .bittorrent branch to have a
common repository to syncronize ourself, and to avoid to repeat the same
work... This should be easier than distributing a lot of patches...
Brian, what do you think about this?
Regards,
-Andrea
Erich Focht wrote:
Bernard, it is good to hear that you make progress. I'm a bit sorry that you
chose the non-general approach. IMO the "one tar method" only works if you
have small images and big memory on the nodes. Systemimager is not only used
with OSCAR and OSCAR images are not allways the standard small images. So
_please_ don't take out the tar pipe from flamethrower/multicast, it would
bring back the problems I had 1 year ago.
Do you already have a feeling from your experiments on how long a bittorrent
deploy (with tar) takes? Did you try deploying the image without the /dev
directory (so without a tar)? Did that take long? I mean: is BT scaling
reasonably with the number of files in the torrent? Just want to make sure
that before making big changes this thing is really worth it.
Thanks,
best regards,
Erich
On Monday 26 December 2005 08:08, Bernard Li wrote:
Okay progress report. I got bittorrent compiled into the initial ramdisk and
it can successfully pull an image tar file from the headnode.
CTorrent seems like a dead project, so I searched for another BitTorrent client
written in C and decided on using LibBT:
http://sourceforge.net/projects/libbt
Looks like this project is still active and at least it works with the latest
"official" BitTorrent client (4.2.2).
It looks like Greg Kurtzer (Warewulf) was trying it out too:
http://sourceforge.net/mailarchive/forum.php?thread_id=6870164&forum_id=34746
The bug Greg pointed out still needs to be fixed in the latest release (1.05),
the -q functionality (once working) would be very useful as we probably want
the clients to automatically quit after a specific timeout period... Here's
the btget argument list:
Usage: btget [options] <torrent>...
Version: 1.05
Options:
-h This help message
-b Disable snub detection
-f<filename> Download file
-q<timeout> Exit after no transfers for timeout secs
-u Interpret <torrent> as a URL from which to fetch torrent
-e<ratio> Exaggerate upload speed [1.0]
-s Seed the file (assume all blocks ok)
-v Verbose (show all peers and transfer rates)
Right now I have to figure out how exactly to write this as another transport
for flamethrower. In a sense the bittorrent method is very similar to
flamethrower/multicast because you need to tar up the image directory prior to
transfer. One difference is that you can't do a tarpipe (because bittorrent is
not a stream). I guess I need to figure out how exactly to replace the
udp-sender/udp-receiver codes... Brian, if you can give me some hints on where
to start, that'd be great. Off the bat, I'll need to re-write a lot of
initrd_source/skel/etc/init.d/functions - I will probably need to do huge
modifications to flamethrower proper as well...
The way I envision BitTorrent working is that each node would keep the tarfile
around even after imaging (such that it can continue to seed when new nodes are
building) - however, this causes an issue when you make modifications to the
image on the head - you'll need to re-tar it and re-distribute the entire image
to each node again... So perhaps we need to figure out a way to use bittorrent
with the image dir instead of making a tarball...
Cheers,
Bernard
--
Brian Elliott Finley
Mobile: 630.631.6621
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Sisuite-devel mailing list
Sisuite-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sisuite-devel
