Hi all,
sorry folks, I'm currently a little overworked, as usual, ... I figured it is
probably time to finally go thru all these emails, ... :-)
Thanks, Mika, for resending the patch. I remember I liked the idea back in the
day you first submitted this back in 2009.
The option parsing is a little fragile, and I assume one would get the best of
both worlds using a small tmpfs by default (100MB or so), to compile like >90%
of the packages with tmpfs, and have big packages fail even during unpacking,
quickly.
I thus propose that I merge a simplified version without the per package size
and caching to get the more robust behavior while running out of tmpfs space.
If it turns out more fancy per package cache sizes are wanted that changes
could be merged in a second step.
Best regards,
René
On 17.08.2011, at 12:30, Mika N wrote:
> Hi,
>
> T2 build script has a nice feature of using TMPFS ramdisk as temporary
> compilation storage to speed up things. However, if certain "big package"
> doesn't fit in TMPFS ramdisk then Build-Target batch job fails to continue.
> In practice most users probably have enough RAM to compile 90% of packages
> but those few huuuuge packages would require several gigas of extra memory
> (gcc, kernel, etc). It would be nice to have T2 automatically revert back to
> ordinary disk usage if TMPFS usage run out of space. This way we can speed up
> 90% of package compilations and use slow disk for those couple huge packages
> only.
>
> Well, here is my proposal for that optimization and diff patch to
> scripts/Build-Pkg file. The patch does following tricks
>
> (1) Checks whether TMPFS configuration option is set (this is normal T2
> config stuff). If TMPFS option is not set then this code change doesn't do
> anything new.
> (2) If TMPFS option was set then script tries to compile the package as
> normally using the TMPFS ramdisk storage if builddir_pkgsize cache file
> indicatates that the pkg build dir space requirement is less than TMPFS size
> (calculated in mega bytes in the current diff patch). If pkg compilation size
> would be more than TMPFS size then script doesnt use TMPFS even when it was
> enabled.
> (3) If pkg compilation failed in TMPFS then script does extra trick and tries
> to re-compile the same package automatically using normal disk space (it does
> this by forcing Build-Pkg script to ignore TMPFS option on the second run)
> (4) If pkg compilation succeeded then script caches the actual use of disk
> space in config/$config/builddir_pkgsize file (build stage and pkg specific
> cache line)
>
> This change caches actual compilation disk space requirements in
> config/$config/builddir_pkgsize file and it is automatically created and
> maintained. One can add manually lines in there as long the line format is as
> follows: stageNumber pkgName compilationDiskRequirementInMegaBytes
>
> For example in my scenario GCC package would have following lines in
> builddir_pkgsize cache file after completing all stages
> 0 gcc 752M
> 1 gcc 882M
> 2 gcc 1520M
> 5 gcc 1520M
>
> As you can see different stages may have different disk requirements, so this
> is why cache file is "per stage per pkg". One can copy this builddir_pkg
> cache file from one configuration to another config folder to re-use cache
> information and to avoid unnecessary "huuuge pkg" TMPFS compilations which
> would fail anyway in out-of-tmpfs-space errors. This saves couple minutes of
> crucial compilation time when those huuuge packages are automatically
> compiled through ordinary disk space even when TMPFS would be enabled.
>
> The patch is a bit brutal when it comes to TMPFS option in t2 configurator
> (normal stuff, this diff doesnt change anything there). It must be configured
> using following syntax (as a sample). TMPFS size in T2 configurator must be
> told in mega bytes and as the first parameter in option line: size=800M,
> nr_inodes=100k. Of course your "mileague may vary" but order of these
> options and syntax should be as shown here.
>
> I hope someone else may find this optimization useful. I have used it quite a
> lot already and it does seem to speed up things at least in my "resource
> contrained laptop". I run T2 compilation environment in vmware image on a
> normal Windows laptop, so my benchmarks probably don't apply to bigger and
> faster native compilation platforms.
>
> Best regards,
> Mika N
>
> <Build-Pkg-TMPFSOpti.patch.txt>-----------------------------------------------------------
>
> If you wish to unsubscribe from this mailing, send mail to
> [email protected] with a subject of: unsubscribe t2
--
René Rebe, ExactCODE GmbH, Jaegerstr. 67, DE-10117 Berlin
DE Legal: Amtsgericht Charlottenburg HRB 105123B, Tax-ID#: DE251602478
Managing Directors: Susanne Klaus, René Rebe
http://exactcode.com | http://t2-project.org | http://rene.rebe.de
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[email protected] with a subject of: unsubscribe t2
