I've had a look at the idea Erich proposes, and I think it makes sense,
and we should implement it.
Here are a couple of things to keep in mind:
* Need to honor expected updateclient behavior, including the
"updateclient.local.exclude" file.
* Need to activate this feature in the same situations as the
current BT code -- only when the user has chosen to use BT as
their transport.
Cheers, -Brian
Thus spake Bernard Li ([EMAIL PROTECTED]):
>Hi Erich:
>
>Do you need this feature included in OSCAR 5.0? If not, then I suggest
>we integrate this after SystemImager 3.7.3 is released unless you think
>you can do this in 1-2 days...
>
>Thanks,
>
>Bernard
>
>> -----Original Message-----
>> From: [EMAIL PROTECTED]
>> [mailto:[EMAIL PROTECTED] On Behalf
>> Of Erich Focht
>> Sent: Monday, May 29, 2006 2:33
>> To: sisuite-devel@lists.sourceforge.net
>> Cc: Brian Elliott Finley
>> Subject: [Sisuite-devel] tardiff for updateclient (or whole
>> deployment?)
>>
>> Hi Brian,
>>
>> now that bittorrent is well included into systemimager I'd
>> like to re-iterate
>> the idea I wrote about in January. The initial email is
>> attached, below.
>>
>> The idea is to use a "tardiff" produced tarball for updating
>> client nodes
>> instead of deploying the entire image. Experiments showed
>> that building a
>> tardiff was 7 to 32 times faster than building the tarball.
>> (tarball build +
>> compress: 130s, tardiff build from compressed tarball: 17s,
>> from uncompressed
>> tarball 3.8s). Additionally its deployment saves a huge
>> amount of bandwidth
>> and it gives one some sort of image management (as tardiffs
>> are small and
>> can be archived easilly).
>>
>> If nothing speaks against this, I'd imagine it would be easy
>> to add a --diff
>> option to si_updateclient which fetches a tardiff instead of
>> doing the entire
>> rsync of the image. Of course, bittorrent deployment could
>> also take into
>> account some image-diff.tar.gz file on top of the original
>> image.tar.gz, with
>> some logic recognising whether a diff is needed at all or not.
>>
>> So, should I proceed with the integration of this? Any
>> thoughts about the
>> approach?
>>
>> Thanks,
>> best regards,
>> Erich
>>
>>
>> ================== initial email: Jan 10 2006
>> ===========================
>>
>> here are some thoughts about updating (with a test install on
>> a 2.4GHz Xeon
>> with RHEL4 in mind):
>>
>> - the full image directory requires 711MB
>> - the tarred up image needs: 640MB
>> - built from the image directory in 31s
>> - compressed image tarfile: 219MB
>> - compress time (gzip) 100s
>> - compress time (gzip -9) 270s
>> - uncompress time 15s
>>
>> For the initial deployment it fully makes sense to work with
>> the gzipped
>> tarfile, this saves a lot of bandwidth and certainly speeds
>> up things even on
>> small clusters.
>>
>> For updates it seems to be a big waste to use the full tar. A
>> tar containing
>> only the differences between the image directory and the
>> initial deployment
>> tarfile would be much faster to deploy. An idea is to use two
>> tars right from
>> the start:
>>
>> image.tar.gz : full archive of the image directory
>> (initial deployment
>> image)
>> image-diff.tar.gz : archive containing the diffs between the
>> image directory
>> and image.tar.gz . We would use only one of these
>> differential archives (not incremental,
>> just base+diff).
>>
>> At the initial deployment time image-diff.tar.gz would be an
>> empty archive.
>>
>> The attached script "tardiff" builds such differential tars.
>> It mainly spends
>> its time with comparing the original image tarfile with the
>> directory content
>> on the filesystem. Actually most time is spent with decompressing the
>> archive. Here are some numbers (with only small differences
>> between image
>> directory and image.tar.gz):
>>
>> - if the tarfile is compressed (image.tar.gz) building
>> image-diff.tar.gz takes
>> 17.6s
>> - if the tarfile is not compressed (image.tar), building
>> image-diff.tar.gz
>> takes 3.8s.
>>
>> So it is really feasible to build (and overwrite)
>> image-diff.tar.gz right
>> before upgrading a cluster. The benefit of using this instead
>> of a full tar is
>> a very much reduced amount of data to be transfered over the
>> network and the
>> fact that unpacking this small archive really touches only
>> the modified files,
>> nothing else.
>>
>> Any thoughts? If this makes sense, I'll try integrating once
>> there is some BT
>> transport available (or the framework around it).
>>
>>
>>
>> -------------------------------------------------------
>> All the advantages of Linux Managed Hosting--Without the Cost
>> and Risk!
>> Fully trained technicians. The highest number of Red Hat
>> certifications in
>> the hosting industry. Fanatical Support. Click to learn more
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&;
>> dat=121642
>> _______________________________________________
>> Sisuite-devel mailing list
>> Sisuite-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/sisuite-devel
>>
--
Brian Elliott Finley
Mobile: 630.631.6621
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sisuite-devel mailing list
Sisuite-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sisuite-devel
