Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
On 07/26/2014 08:50 PM, Kir Kolyshkin wrote: On 07/26/2014 12:45 PM, Kevin Holly wrote: On 23/07/14 09:34, Kir Kolyshkin wrote: I just talked with a L3 Supporter of SolusVM and they told me that this seems to be a well known vzctl bug. Great, can they point out to a bug in bugzilla.openvz.org? What SolusVM basically does is this on creation of the container: https://ezcrypt.it/ag9n#QcRKRX8DIFdFKJv6ZNFahTSt Thanks for providing this! Lots of weird/stupid things overall... But let's stay on topic. Here is the first relevant line: vzctl create 134 --layout ploop --diskspace 262144000K:262144000K [...] Here they set diskspace to 250G on creation. And this is the second relevant line: vzctl set 134 --diskspace 262144000K:262144000K --diskinodes 131072000:131072000 --save Here they resize it with 250G diskspace and 131072000 of diskinodes. Now, ext4 reserve one inode per 16KB of diskspace on filesystem creation (you can think of it as a heuristic that an average file is 16KB of size), and there is no way to change a number of inodes on an existing filesystem. Well, there is one -- you need to resize the filesystem itself. As vzctl is told that 131072000 of diskinodes are required, it resizes the file system to the size sufficient to hold the requested number of inodes, which is 131072000*16K, or 2TB. Then, as diskspace is set to 250G, vzctl steals the rest of the space by using an invisible hidden (empty) balloon file. So, while the internals are tricky, the result is exactly as requested, i.e. a filesystem with (about) 250 GB of diskspace, but with 131072000 inodes. Now, if I can, I'd say that the number of inodes requested is way too high for the given disk space, unless it's really going to be used to hold that high number of small (about 2K on average) files. I would like to add that previously SolusVM was passing unlimited to --diskinodes for vzctl set, which transforms to LONG_MAX (9223372036854775807 (2^63 - 1) on an x86_64 architecture) which is definitely way too high and resulted in a ploop image creation error (size too large or something like this). So, it seemed like a regression in vzctl 4.7 (compared to vzctl 4.6 which just ignored diskinodes value for ploop-based CTs), but is in fact a bug in SolusVM which was uncovered by a new vzctl feature. Now, instead of fixing it, apparently they just changed unlimited to 131072000, which is not as insane as unlimited but still not sane for smaller CTs (a sane default would be to omit diskinodes altogether) And this is the reinstallation process of the container: https://ezcrypt.it/cg9n#sHQJeiV7Y6bl6zyBBjS60oLN Where the config looks exactly like this: https://ezcrypt.it/dg9n#g2Ico8AknpAtJbAcxkbBQDub This adds nothing new to the picture and is in line with that was seen before. Now, back to vzctl bug that they say exist in versions newer than 4.6.1. This diskinodes handing for ploop was added in vzctl 4.7, before that diskinodes value was ignored for ploop. Hope that helps. shameless plug By the way, we accept donations now, so if you feel like helping us back, too, feel free: http://openvz.livejournal.com/48757.html /shameless plug Kir. ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 23/07/14 09:34, Kir Kolyshkin wrote: I just talked with a L3 Supporter of SolusVM and they told me that this seems to be a well known vzctl bug. What SolusVM basically does is this on creation of the container: https://ezcrypt.it/ag9n#QcRKRX8DIFdFKJv6ZNFahTSt And this is the reinstallation process of the container: https://ezcrypt.it/cg9n#sHQJeiV7Y6bl6zyBBjS60oLN Where the config looks exactly like this: https://ezcrypt.it/dg9n#g2Ico8AknpAtJbAcxkbBQDub - -- Best regards Kevin Holly - r...@hallowe.lt - http://hallowe.lt/ -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJT1AVjAAoJELAaqP3QtzpMU/4H/24DuHnv8winheyx9TBT9BWD cDCF1Xj2lG6Q5nvdJSNx/qL+6ZODi8lxzFRHvQXTetBBCLxyLFM8LxFGTfYe4Sdb uPMqjT2Ukva93VtOMtMGMu8nOtBN6/p8z9rL+y6HQ32EwyeiHKznlKsGr0QGm6zd omPyvb2FdXHlTwxpj9bfjsYZsj8o6AGc8Bezb8Luw2SgBlYLDkCJRfZDMOsaHPov kP6B3yHuoJJDrxvDvpYGqVNGethQZK+bkcSDbIPls8ZW6LeMQeykEmE1vhbrXGYa UzmhGEGEjmsmERQ5n6bG/xSQmBLSeJsG+0Obu1A3QgHq/3tIP8LnYK61YHex1kM= =DWo/ -END PGP SIGNATURE- ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26/07/14 21:45, Kevin Holly wrote: Oh, and downgrading vzctl to version vzctl-4.6.1-1.x86_64 helped for now... - -- Best regards Kevin Holly - r...@hallowe.lt - http://hallowe.lt/ -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJT1BA2AAoJELAaqP3QtzpM//oH/3CctpINGDFRqGYWnI8ArKwJ 4sACnKnBYWqEu9hayu5Akaqu7BnRBZlPcXdMekoIpEcQrvt2FGBvp2XyXNLJ64hY HRwLWGuX5K/AAFx6gAQ0bTD4DSe72Nid3ltNsxM1L7CcsED6J/vOSZw9aPznaIlf CRtBUMr2QrLMcWfS0Jk9t6vMSqpid43X43CE3VOx9nEbjiOLgY0WatWHHjXqw2Ep 4Gpy0rEXmB2qFWCAwn2HsYKc+5rVGSUS4vn7UtJaPmRYQSr84ylEF4SWK4FTU0q9 OPcmou4MdqPcbbEc9rmewTQ2QK34wrYV5NvaX1gyNDBhH+TeBVoxZaK0labEeAM= =RL4O -END PGP SIGNATURE- ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
On 07/26/2014 12:45 PM, Kevin Holly wrote: On 23/07/14 09:34, Kir Kolyshkin wrote: I just talked with a L3 Supporter of SolusVM and they told me that this seems to be a well known vzctl bug. Great, can they point out to a bug in bugzilla.openvz.org? What SolusVM basically does is this on creation of the container: https://ezcrypt.it/ag9n#QcRKRX8DIFdFKJv6ZNFahTSt Thanks for providing this! Lots of weird/stupid things overall... But let's stay on topic. Here is the first relevant line: vzctl create 134 --layout ploop --diskspace 262144000K:262144000K [...] Here they set diskspace to 250G on creation. And this is the second relevant line: vzctl set 134 --diskspace 262144000K:262144000K --diskinodes 131072000:131072000 --save Here they resize it with 250G diskspace and 131072000 of diskinodes. Now, ext4 reserve one inode per 16KB of diskspace on filesystem creation (you can think of it as a heuristic that an average file is 16KB of size), and there is no way to change a number of inodes on an existing filesystem. Well, there is one -- you need to resize the filesystem itself. As vzctl is told that 131072000 of diskinodes are required, it resizes the file system to the size sufficient to hold the requested number of inodes, which is 131072000*16K, or 2TB. Then, as diskspace is set to 250G, vzctl steals the rest of the space by using an invisible hidden (empty) balloon file. So, while the internals are tricky, the result is exactly as requested, i.e. a filesystem with (about) 250 GB of diskspace, but with 131072000 inodes. Now, if I can, I'd say that the number of inodes requested is way too high for the given disk space, unless it's really going to be used to hold that high number of small (about 2K on average) files. And this is the reinstallation process of the container: https://ezcrypt.it/cg9n#sHQJeiV7Y6bl6zyBBjS60oLN Where the config looks exactly like this: https://ezcrypt.it/dg9n#g2Ico8AknpAtJbAcxkbBQDub This adds nothing new to the picture and is in line with that was seen before. Now, back to vzctl bug that they say exist in versions newer than 4.6.1. This diskinodes handing for ploop was added in vzctl 4.7, before that diskinodes value was ignored for ploop. Hope that helps. shameless plug By the way, we accept donations now, so if you feel like helping us back, too, feel free: http://openvz.livejournal.com/48757.html /shameless plug Kir. ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
On 07/26/2014 01:31 PM, Kevin Holly wrote: On 26/07/14 21:45, Kevin Holly wrote: Oh, and downgrading vzctl to version vzctl-4.6.1-1.x86_64 helped for now... For the exact reason I explained in the previous email -- it just ignores diskinodes. Not sure why SolusVM guys can't figure it out (or explain to customers), definitely not rocket science ;) Kir ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Kir, On 23/07/14 00:25, Kir Kolyshkin wrote: I failed to reproduce your problem locally: Okay, then I have to dig further and see where the problem is. Final question, just curious -- are you paying to SolusVM? If yes, why don't you use their support to help with your issues, relying on a free help from OpenVZ community and deveopers instead? I thought that this may be an OpenVZ bug or that I get more useful support as SolusVM guys are not really helpful when such things are happening, but I'll try to contact and talk with them and lets see where this goes. Although, thanks for your help so far. - -- Best regards Kevin Holly - r...@hallowe.lt - http://hallowe.lt/ -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTz14fAAoJELAaqP3QtzpMndsH+gLgVJvj3eWB/jLt8bCg6i7W tny+w5fuKA6A/WLnCpdaNZhnJ/dP98RAPsR3HqJe2bvgzggWoEm3YyENROjumKQZ 9Q9DrjSCnogBNs9wtK2cUZ4wmdqKQBwer327rgaOS51tXFWCO2lMC1p9WncNf95J qEIGEGYLT26Ul3X4CH26Aroq2R+hMj3ioEeDYxHjpXAgjZCyg4bLWGoOGbdx7eif Mg6piDB/9VtgpyFyeei9UVTOqzy9R0mAxLvx2h+8eDjkvp8rnh/9wSSAAMGfNGSk 66J9EabpDtZmkpaJh8yGIfi8nE8klhgD09XcU0/mQzbm1A2pBWIZAGTv2cJDe74= =/rQF -END PGP SIGNATURE- ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi again, I just tried to reinstall this container... The log of that can be found here: https://ezcrypt.it/Hd9n#SiP14ajsc8sV0SLiGm1diEc1 There is not much difference with the ploop filesystem now. Same size (2TB but only 219GiB/235GB available) and ploop info on the DiskDescriptor.xml also reports 1k-blocks 229233640 (wrong size). On 22/07/14 21:27, Kevin Holly wrote: Hi, I have a customer complaining about his filesystem being too small (product FS size: 250GB, his FS size: 219GiB / 235GB). After further debugging this issue I am now here: Creation of the containers filesystem (I can provide more logs, by default I have LOG_LEVEL=7 and VERBOSE=1 2014-06-16T19:20:02+0200 vzctl : CT 2671 : Creating image: /vz/private/2671.tmp/root.hdd/root.hdd size=262144000K 2014-06-16T19:20:02+0200 : Creating delta /vz/private/2671.tmp/root.hdd/root.hdd bs=2048 size=524288000 sectors v2 2014-06-16T19:20:02+0200 : Adding snapshot {5fbaabe3-6958-40ff-92a7-860e329aab41} 2014-06-16T19:20:02+0200 : Storing /vz/private/2671.tmp/root.hdd/DiskDescriptor.xml 2014-06-16T19:20:02+0200 : Opening delta /vz/private/2671.tmp/root.hdd/root.hdd 2014-06-16T19:20:02+0200 : Adding delta dev=/dev/ploop16059 img=/vz/private/2671.tmp/root.hdd/root.hdd (rw) 2014-06-16T19:20:02+0200 : Running: parted -s /dev/ploop16059 mklabel gpt mkpart primary 1048576b 268434407423b 2014-06-16T19:20:02+0200 : Running: mkfs -t ext4 -j -b4096 -Elazy_itable_init,resize=4294967295 -Jsize=128 -i16384 /dev/ploop16059p1 2014-06-16T19:20:03+0200 : Running: /sbin/tune2fs -ouser_xattr,acl -c0 -i0 /dev/ploop16059p1 2014-06-16T19:20:03+0200 : Creating balloon file .balloon-c3a5ae3d-ce7f-43c4-a1ea-c61e2b4504e8 2014-06-16T19:20:03+0200 : Mounting /dev/ploop16059p1 at /vz/private/2671.tmp/root.hdd/root.hdd.mnt fstype=ext4 data='' 2014-06-16T19:20:03+0200 : Unmounting device /dev/ploop16059 2014-06-16T19:20:03+0200 : Opening delta /vz/private/2671.tmp/root.hdd/root.hdd 2014-06-16T19:20:03+0200 : Adding delta dev=/dev/ploop16059 img=/vz/private/2671.tmp/root.hdd/root.hdd (rw) 2014-06-16T19:20:03+0200 : Mounting /dev/ploop16059p1 at /vz/root/2671 fstype=ext4 data='balloon_ino=12,' solus-ed-ch01:~# vzctl exec 2671 df -B1 / Executing command: df -B1 / Filesystem 1B-blocksUsedAvailable Use% Mounted on /dev/ploop16059p1 234735247360 86872219648 134441467904 40% / solus-ed-ch01:~# vzctl exec 2671 df -h / Executing command: df -h / Filesystem Size Used Avail Use% Mounted on /dev/ploop16059p1 219G 81G 126G 40% / solus-ed-ch01:~# gdisk -l /dev/ploop16059 GPT fdisk (gdisk) version 0.8.10 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /dev/ploop16059: 4194304000 sectors, 2.0 TiB Logical sector size: 512 bytes Disk identifier (GUID): FA4B229E-10F7-4583-8A4F-C7B99A34945D Partition table holds up to 128 entries First usable sector is 34, last usable sector is 4194303966 Partitions will be aligned on 2048-sector boundaries Total free space is 4029 sectors (2.0 MiB) Number Start (sector)End (sector) Size Code Name 12048 4194301951 2.0 TiB 0700 primary So here you can see that the ploop device got provisioned with 2TB ploop filesystem size and the container only sees 234735247360 Byte (235 GB / 219 GiB). Can someone help me debugging this issue further? This host node had no crashes yet, all other containers are working completely fine. It's running the following (patched) kernel version: solus-ed-ch01:~# uname -a Linux solus-ed-ch01 2.6.32-042stab090.2 #1 SMP Wed May 21 19:25:03 MSK 2014 x86_64 x86_64 x86_64 GNU/Linux solus-ed-ch01:~# kcare-uname -a Linux solus-ed-ch01 2.6.32-042stab092.2 #1 SMP Wed May 21 19:25:03 MSK 2014 x86_64 x86_64 x86_64 GNU/Linux Thanks in advance for anyone trying to help! Best regards Kevin Holly - r...@hallowe.lt - http://hallowe.lt/ ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users - -- Best regards Kevin Holly - r...@hallowe.lt - http://hallowe.lt/ -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTzts/AAoJELAaqP3QtzpMYCwH/R3ZQOMs/Wr/vf6aVrQs1ByP JVCqoQaHI7j3SX6T12pKBh3GqRThTZVoLHeOFnGNVR7Dc3YXgJgAmscJUQ1P6fVF x6+5A1OVLrZisvmyLGmL+fFL9kJlBsmMhc1DW7zie31FFcPfMlpE6l6ypC6NW/de qPN8Pf2cDh4hnr44SKKAs6gLsPl140S2x1VDyVKQH5L97wDUHAeMgNt35G1dKXLw l6A65fSB1MZGY8vdqDvukJsePFg5j6Vr4vSemEv5IIQzbD52/oHwdjqMzhOVxu9H kd2SepI4xOdmhCMugNiZ8bUp7xH8n5qc+QgiJ3K1mFHzE97d/5/5D4vtZS+3XLk= =4Xga -END PGP SIGNATURE- ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] Ploop filesystem provisioned incorrectly (wrong size)
On 07/22/2014 02:44 PM, Kevin Holly wrote: Hi again, I just tried to reinstall this container... The log of that can be found here: https://ezcrypt.it/Hd9n#SiP14ajsc8sV0SLiGm1diEc1 The only strange thing I see is Killing container, perhaps you use an incorrect sequence for vzctl set --userpasswd. Note you do not need to start a container to use it. But this is not related, and I guess SolusVM guys are to blame. I failed to reproduce your problem locally: # vzctl create 345 --ostemplate ubuntu-12.04-x86_64-minimal --layout ploop --diskspace 2T ... vzctl start 345 ... # vzctl exec 345 df -h Executing command: df -h Filesystem Size Used Avail Use% Mounted on /dev/ploop11994p1 2.0T 370M 1.9T 1% / ... # ploop info /vz/private/345/root.hdd/DiskDescriptor.xml resource Size Used 1k-blocks 2113785776 378220 inodes 134217728 12599 Having said that, I know a way to get to the result you are seeing. If we run vzctl set 345 --diskspace 256G --save (or perhaps vzctl set --applyconfig with config containing DISKSPACE parameter) it will result in something that you see -- a partition of 2G with a filesystem of about 256G. The reason is when shrinking FS ploop uses a hidden balloon rather than a real resize -- it is much more efficient this way. So, can you please provide a sequence of commands to lead to the problem (starting from vzctl create or earlier)? I understand you are using SolusVM, but in order to demonstrate that there is a issue in OpenVZ we should try to eliminate extra variables. Final question, just curious -- are you paying to SolusVM? If yes, why don't you use their support to help with your issues, relying on a free help from OpenVZ community and deveopers instead? Regards, Kir. There is not much difference with the ploop filesystem now. Same size (2TB but only 219GiB/235GB available) and ploop info on the DiskDescriptor.xml also reports 1k-blocks 229233640 (wrong size). On 22/07/14 21:27, Kevin Holly wrote: Hi, I have a customer complaining about his filesystem being too small (product FS size: 250GB, his FS size: 219GiB / 235GB). After further debugging this issue I am now here: Creation of the containers filesystem (I can provide more logs, by default I have LOG_LEVEL=7 and VERBOSE=1 2014-06-16T19:20:02+0200 vzctl : CT 2671 : Creating image: /vz/private/2671.tmp/root.hdd/root.hdd size=262144000K 2014-06-16T19:20:02+0200 : Creating delta /vz/private/2671.tmp/root.hdd/root.hdd bs=2048 size=524288000 sectors v2 2014-06-16T19:20:02+0200 : Adding snapshot {5fbaabe3-6958-40ff-92a7-860e329aab41} 2014-06-16T19:20:02+0200 : Storing /vz/private/2671.tmp/root.hdd/DiskDescriptor.xml 2014-06-16T19:20:02+0200 : Opening delta /vz/private/2671.tmp/root.hdd/root.hdd 2014-06-16T19:20:02+0200 : Adding delta dev=/dev/ploop16059 img=/vz/private/2671.tmp/root.hdd/root.hdd (rw) 2014-06-16T19:20:02+0200 : Running: parted -s /dev/ploop16059 mklabel gpt mkpart primary 1048576b 268434407423b 2014-06-16T19:20:02+0200 : Running: mkfs -t ext4 -j -b4096 -Elazy_itable_init,resize=4294967295 -Jsize=128 -i16384 /dev/ploop16059p1 2014-06-16T19:20:03+0200 : Running: /sbin/tune2fs -ouser_xattr,acl -c0 -i0 /dev/ploop16059p1 2014-06-16T19:20:03+0200 : Creating balloon file .balloon-c3a5ae3d-ce7f-43c4-a1ea-c61e2b4504e8 2014-06-16T19:20:03+0200 : Mounting /dev/ploop16059p1 at /vz/private/2671.tmp/root.hdd/root.hdd.mnt fstype=ext4 data='' 2014-06-16T19:20:03+0200 : Unmounting device /dev/ploop16059 2014-06-16T19:20:03+0200 : Opening delta /vz/private/2671.tmp/root.hdd/root.hdd 2014-06-16T19:20:03+0200 : Adding delta dev=/dev/ploop16059 img=/vz/private/2671.tmp/root.hdd/root.hdd (rw) 2014-06-16T19:20:03+0200 : Mounting /dev/ploop16059p1 at /vz/root/2671 fstype=ext4 data='balloon_ino=12,' solus-ed-ch01:~# vzctl exec 2671 df -B1 / Executing command: df -B1 / Filesystem 1B-blocksUsedAvailable Use% Mounted on /dev/ploop16059p1 234735247360 86872219648 134441467904 40% / solus-ed-ch01:~# vzctl exec 2671 df -h / Executing command: df -h / Filesystem Size Used Avail Use% Mounted on /dev/ploop16059p1 219G 81G 126G 40% / solus-ed-ch01:~# gdisk -l /dev/ploop16059 GPT fdisk (gdisk) version 0.8.10 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /dev/ploop16059: 4194304000 sectors, 2.0 TiB Logical sector size: 512 bytes Disk identifier (GUID): FA4B229E-10F7-4583-8A4F-C7B99A34945D Partition table holds up to 128 entries First usable sector is 34, last usable sector is 4194303966 Partitions will be aligned on 2048-sector boundaries Total free space is 4029 sectors (2.0 MiB) Number Start (sector)End (sector) Size Code Name 12048 4194301951 2.0 TiB