Package: installation-report
Worth knowing if you use VirtualBox with Debian9 guests? : The rubyv3 guest is a fresh install of Debian9 from a CD including graphics and intended as a workstation. The rubyv4 guest is a fresh install of Debian9 from a network install CD with no optional software and intended to be built as a server. The rubyv5 guest is a fresh install of Ubuntu Server with no optional software. All are Virtualbox virtual machines. The underlying hardware OS is the same Debian9 with a 6TB array of five hard disks. File Sizes: rubyv3(debian w/s) rubyv4(debian srv) rubyv5(ubuntu srv) System use inside VM 3,071,064,000 1,189,448,000 1,554,552,000 reported by 'ls -l' .vdi size 4,942,987,264 109,515,374,592 4,632,608,768 .ova size 1,569,787,904 110,322,772,480 4,482,602,496 Why the difference in the .vdi and .ova file sizes for rubyv4, as compared to the files for rubyv3 and ruby5? All the file sizes for rubyv3 are reasonable, but ... ... requiring a 110GB .ova backup file for a 1.2GB install for rubyv4 is not! The reason for this appears to lie in the Debian install process and its interaction with Virtualbox. rubyv3 was installed onto an ext4 partition; rubyv4 was installed onto an encrypted LVM partition where the dynamic 110GB .vdi file was zeroed(with random numbers?) to its full size. Ubuntu server, using an encrypted LVM, dynamic 100GB partition layout, does not do the same(no zeroing of free space) but the compression algorithm achieves next to nothing in producing the .ova file. The desire to completely 'zero' a hardware disk before installing a new operating system is sensible but doing the same to a dynamic virtual disk that is encrypted is questionable. Compacting Virtualbox .vdi files: > dd if=/dev/zero of=zerofillfile status=progress bs=1M until the disk fills, then > rm zerofillfile and all free space is zeroed. Do not use 'if=/dev/urandom' in the 'dd' command!! > VBoxManage modifyhd <path/file.vdi> --compact https://unix.stackexchange. <https://unix.stackexchange.com/questions/85865/trim-with-lvm-and-dm-crypt> cat /etc/crypttab issue_discards = 1 sda3_crypt UUID=[...] none luks,discard cat /etc/lvm/lvm.conf # [...] devices { # [ ... ] issue_discards = 1 # [ ... ] } # [...] To test the setup I just did 'sudo fstrim -v /' which resulted in /: [...] bytes were trimmed Doing this again resulted in /: 0 bytes were trimmed which seems to make sense and indicated that TRIM seems to work.However then I did this test: dd if=/dev/urandom of=tempfile count=100 bs=512k oflag=direct Note 'urandom' https://www.virtualbox.org/ <https://www.virtualbox.org/manual/ch08.html#vboxmanage-modifyvdi> 'The --compact option, can be used to compact disk images, i.e. remove blocks that only contains zeroes[and not the output of 'urandom'!!].' 'compacting is currently only available for VDI images. A similar effect can be achieved by zeroing[not urandom!!] out free blocks and then cloning the disk to any other dynamically allocated format.' In my tests I failed to achieve a .vdi compaction, including by cloning. Further tests : File Sizes: rubyv4(debian srv) 8GB virtualdisk System use inside VM 876,928,000 reported by 'ls -l' .vdi size 8,498,708,480 .ova size 8,508,642,304 resize to 20GB using VBoxManage modifyhd ruby4.vdi --resize 20000 Before starting 8,498,708,480 reported as 19.53GB in VB After run and shutdown 8,498,708,480 File Sizes: rubyv6(debian srv) 8GB virtualdisk Debian install zeroing of free disk space halted by escaping the zero procedure System use inside VM 867,948,000 reported by 'ls -l' .vdi size 1,809,842,176 .ova size 1,636,825,000 resize to 20GB using VBoxManage modifyhd ruby6.vdi --resize 20000 Before starting 1,809,842,176 reported as 19.53GB in VB After run and shutdown 1,809,842,176 which is what it should be Conclusions : 1:After installing Debian9 to a large dynamic Virtualbox virtual disk, the .vdi file goes to full size as the disk is zeroed (with random numbers?) and will not compress into the .ova file or clone. 2: Best to escape the Debian install zeroing of free space or use Ubuntu server. 3: If free space zeroing is required, use the dd command in the running server, but it will take the .vdi file up to full size. 4: Dynamic file expansion will negate the zeroing, unless the zeroing is repeated. 5: Virtualbox .vdi file compaction still seems problematic. Recomendation : Add a clear warning in the Debian install process about zeroing free space in large Virtualbox virtual disks. The resulting .vdi file bloat is, for current purposes, irreversible. ####
