Re: [Users] OpenVZ and ZFS excellent experience
Hello, all! Thank you for feedback! Kirill, you are absolutely right and this issue mentioned in my comparison table https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/OpenVZ_containers_on_zfs_filesystem.md But there are some progress at this field there https://github.com/zfsonlinux/zfs/issues/2922 and there https://github.com/zfsonlinux/zfs/pull/2577 This issue can be solved with using ZVOL's instead ZFS native volumes. There are my manual about this: https://github.com/pavel-odintsov/OpenVZ_ZFS/raw/master/openvz_and_zfs_zvol_ext4.pdf (sorry, it's only in russian but you feel free to use Google Translate :). On Mon, Jan 12, 2015 at 8:55 AM, Kirill Korotaev d...@parallels.com wrote: BTW, Pavel one issue which you or others might consider and test well before moving to ZFS: 2nd level (i.e. CT user) disk quotas. One will have to emulate Linux quota APIs and quota files for making this work. e.g. some apps like CPanel call quota tools directly and depending on OS installed in container these quota tools expect slightly different Linux quota behavior/APIs. In the past there was a lot of problems with that and we even emulated quota files via /proc. So be warned. On 11 Jan 2015, at 23:16, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello! Because your question is very big I will try to answer in multiple blocks :) --- My disk space issue. 24GB is a wasted space from only one container :) Total wasted space per server is about 900Gb and it's really terrible for me. Why? Because I use server SSD with hardware RAID array's and cost per TB is cosmic! I want to give more fast space to customers instead wasting it! :) --- What I want. What I want from OpenVZ community? I want share my positive experience and build strong community of runners ZFS together with OpenVZ :) Well, I still have one question to openvz team related with behavior of vzctl which is important for ZFS (and another fs too): https://bugzilla.openvz.org/show_bug.cgi?id=3166 --- License issues of ZFS. License issues is not an critical because installing of ZFS is straightforward and do not require any deep integration to system or kernel and work on almost any kernel. And we can call zfs tools (zpool, zfs) without any problems with CDDL license of ZFS. But we can't link to libzfs and fortunately we do not need this. --- Ploop/ext4 vs ZFS ploop builded on top of ext4 and I compare ZFS with ploop and ext4 and many issues notified in my table related with features of both them. Obviously, it's completely incorrect to compare ploop (block level mapper device) with classic filesystem. --- Conclusion Globally, my speech is not related with ZFS itself. It's about storage system for containers. It's most important part of any virtualization technology. Ploop is real revolution in containers world! I really appreciate developers of ploop and love them (and will be happy to bring some beer to they) :) But ploop is not a final step of storage system for containers. And it have big problems described here: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/ploop_issues.md and everybody should know this issues. Ignoring this issues will produce complete data loss on important data! ZFS is not ideal filesystem for containers too! It lacks of very big amount of very important features but it's more reliable and featureful than ploop/ext4 :) Thank you! On Sun, Jan 11, 2015 at 9:57 PM, Scott Dowdle dow...@montanalinux.org wrote: Greetings, - Original Message - And I checked my containers with 200% disk overuse from first message and got negative result. 24Gb of wasted space is not related with cluster size issue. Yeah, but 24GB is a long way off from your original claim (if I remember correctly) of about 900GB... but those probably aren't comparing the same things anyway. I'm lost... because ploop and zfs are not, so far as I can tell, competing filesystems on the same level. zfs is competes with other filesystems like ext4 or xfs... whereas for OpenVZ, so far as I know, there isn't a disk-file-as-disk competitor. Given the popularity and stability of the current qcow2 format popularized by KVM/qemu... and the large number of tools compatible with qcow2 (see libguestfs)... I'm wondering if it would be valuable to add qcow2 support to OpenVZ? You are currently using zfs with OpenVZ, correct? And you didn't have to modify any of the OpenVZ tools in order to do so, correct? If that is the case, what is it you want from the OpenVZ project with regards to zfs? So far as I'm concerned the license incompatiblity with the zfs/openzfs makes it where it can not be distributed with stuff licensed under the GPL... so I don't really see a way for OpenVZ to ever ship a zfs-enabled kernel... but yeah, if needed they could add support for it in the tools if that makes sense. I'm unclear on what you are looking
Re: [Users] OpenVZ and ZFS excellent experience
Greetings, - Original Message - License issues of ZFS. License issues is not an critical because installing of ZFS is straightforward and do not require any deep integration to system or kernel and work on almost any kernel. OpenZFS and zfsonline people claim that it is perfectly valid to ship zfs binary kernel modules, see: http://open-zfs.org/wiki/Talk:FAQ Unless I misunderstood, they also say there that ZFS code can be merged into the Linux source tree... but that distributing a binary built from it would be a no-no. They can say a lot of things but what really matters is how the distros behave. So far almost no distros include ZFS kernel modules and related support packages... and (I believe) the reason is that they want to mitigate risk. Quite a few ship the fuse-based ZFS stuff. I believe the small handful of distros that do include ZFS support via kernel modules are located outside of the US. With regards to OpenVZ it mostly matters what Red Hat does and clones. I know Proxmox is a huge Debian fan... does Debian offer ZFS kernel modules and if not, why not? How about Proxmox VE? You can link to libzfs. As example, see grub code. Grub is GPL and they link with libzfs. Do I miss something? Again, are distros shipping grub2 with or without ZFS support? I ask because I think I know the answer but I'm not sure. If ZFS as kernel module (since fuse isn't available at boot is it?) isn't provided by a distro, grub2 support for it probably isn't offered either. I'm just guessing here. TYL, -- Scott Dowdle 704 Church Street Belgrade, MT 59714 (406)388-0827 [home] (406)994-3931 [work] ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
Hello! I can't find any info about linking :( But I found big article from ZoL team: http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue Will be fine if OpenVZ command can add ZFS into standard shipment :) On Mon, Jan 12, 2015 at 10:00 AM, Dietmar Maurer diet...@proxmox.com wrote: License issues of ZFS. License issues is not an critical because installing of ZFS is straightforward and do not require any deep integration to system or kernel and work on almost any kernel. OpenZFS and zfsonline people claim that it is perfectly valid to ship zfs binary kernel modules, see: http://open-zfs.org/wiki/Talk:FAQ And we can call zfs tools (zpool, zfs) without any problems with CDDL license of ZFS. But we can't link to libzfs and fortunately we do not need this. You can link to libzfs. As example, see grub code. Grub is GPL and they link with libzfs. Do I miss something? -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
Unless I misunderstood, they also say there that ZFS code can be merged into the Linux source tree... but that distributing a binary built from it would be a no-no. They claim distributing as binary module is no problem! They have split the code into spl (Solaris porting Layer), and a separate zfs module which use the SPL interface. That make it really hard to claim that zfs is derived work from Linux. They can say a lot of things but what really matters is how the distros behave. So far almost no distros include ZFS kernel modules and related support packages... and (I believe) the reason is that they want to mitigate risk. Quite a few ship the fuse-based ZFS stuff. I believe the small handful of distros that do include ZFS support via kernel modules are located outside of the US. see: http://warpmech.com/?news=myth-busting-series-zfs-on-linux-has-license-problems That article claims that Lawrence Livermore National Lab already ships binary zfs modules to customers. With regards to OpenVZ it mostly matters what Red Hat does and clones. I know Proxmox is a huge Debian fan... does Debian offer ZFS kernel modules and if not, why not? How about Proxmox VE? Proxmox is working on that. You can link to libzfs. As example, see grub code. Grub is GPL and they link with libzfs. Do I miss something? Again, are distros shipping grub2 with or without ZFS support? Proxmox VE will ship grub with zfs support. But I think further distros will follow soon. ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
I know Proxmox is a huge Debian fan... does Debian offer ZFS kernel modules and if not, why not? How about Proxmox VE? Besides, I would like to improve support for more storage types on OpenVZ. I think direct support for zfs, rbd, dm-thin would be great (snaphshot, clone). But for me the current OpenVZ status is a bit unclear, because of the announced move to virtuozzo-core. I guess that means the OpenVZ vzctl code will be completely replaced? And when can we expect a first release? ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
And I checked my containers with 200% disk overuse from first message and got negative result. 24Gb of wasted space is not related with cluster size issue. ./ploop_gramentation_checker.py /vz/private/41507/root.hdd/root.hdd We count 43285217280 bytes We count 6079506655 zero bytes We count 37205710625 non zero bytes We have 14.045226 % of space lost due to ploop fragmentation On Sun, Jan 11, 2015 at 7:45 PM, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello, folks! I read your message again and found suggestion about decreasing block size of ploop. But unfortunately it's not possible with vzctl in any ways. We can do it only with direct call of ploop. Because I can't change block size or recreate VE with another block size I tried to do some research about space lost with current block size. I wrote tool for checking amount of wasted space in ploop: https://gist.github.com/pavel-odintsov/d5c37316e538908e0f01 Sorry, I'm not a good pythoner and any feedback/hate/complains/optimizations about this code are welcome. Everyone can check how many space it can save if reduce ploop block size. Some data from me: We count 5276434432 bytes We count 1360051876 zero bytes We count 3916382556 non zero bytes We have 25.775965 % of space lost due to ploop fragmentation We count 1105199104 bytes We count 509808990 zero bytes We count 595390114 non zero bytes We have 46.128249 % of space lost due to ploop fragmentation On Sat, Jan 10, 2015 at 5:50 PM, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello! Thank you! I will contact with you out off list. On Sat, Jan 10, 2015 at 4:44 PM, Kirill Korotaev d...@parallels.com wrote: Pavel, it’s impossible to analyze it just by `du` and `df` output, so please give me access if you want me to take a look into it. (e.g. if I would create 10 million of 1KB files du would show me 10GB while ext4 (and most other file systems) would allocate 40GB in reality assuming 4KB block size) Thanks, Kirill On 10 Jan 2015, at 00:54, Pavel Odintsov pavel.odint...@gmail.com wrote: Thank you, Kirill! I am grateful for your answer! I reproduced this issue specially for you on one container with 2.4 times (240% vs 20%) overuse. I do my tests with current vzctl and ploop 1.12.2 (with fixed http://bugzilla.openvz.org/show_bug.cgi?id=3156). Please check this gist: https://gist.github.com/pavel-odintsov/b2162c0f7588bb8e5c15 I can't describe this behavior without complying on ext4 data But I I will be very happy if you fix it :) On Sat, Jan 10, 2015 at 12:29 AM, Kirill Korotaev d...@parallels.com wrote: On 09 Jan 2015, at 21:39, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello, everybody! Do somebody have any news about ZFS and OpenVZ experience? Why not? Did you checked my comparison table for simfs vs ploop vs ZFS volumes? You should do it ASAP: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/openvz_storage_backends.md Still not interesting? For example if you have 5Tb disk array (used up to 90%) and using ploop now you lose about 800GB of disk space! Well, AFAIR we simply have a threshold that ploop is not compacted until it’s size is 20% bigger then it should be… Also you can try smaller ploop block size. Anyway, my point is that it has nothing to do with ext4 metadata as stated in your table. This data is from real HWN with few hundreds of containers. I have excellent experience and very good news about ZFS! ZFS on Linux team will add very important feature, linux quota inside container (more details here https://github.com/zfsonlinux/zfs/pull/2577 But still no news about ZFS from OpenVZ team (and even from Virtuozza Core) and we can work separately :) Fortunately, we do not need any support from vzctl and can use raw vzctl with some lightweight manuals from my repo: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/OpenVZ_containers_on_zfs_filesystem.md I collected all useful information here https://github.com/pavel-odintsov/OpenVZ_ZFS Stay tuned! Join to us! -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- Sincerely yours, Pavel Odintsov -- Sincerely yours, Pavel Odintsov -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
Greetings, - Original Message - And I checked my containers with 200% disk overuse from first message and got negative result. 24Gb of wasted space is not related with cluster size issue. Yeah, but 24GB is a long way off from your original claim (if I remember correctly) of about 900GB... but those probably aren't comparing the same things anyway. I'm lost... because ploop and zfs are not, so far as I can tell, competing filesystems on the same level. zfs is competes with other filesystems like ext4 or xfs... whereas for OpenVZ, so far as I know, there isn't a disk-file-as-disk competitor. Given the popularity and stability of the current qcow2 format popularized by KVM/qemu... and the large number of tools compatible with qcow2 (see libguestfs)... I'm wondering if it would be valuable to add qcow2 support to OpenVZ? You are currently using zfs with OpenVZ, correct? And you didn't have to modify any of the OpenVZ tools in order to do so, correct? If that is the case, what is it you want from the OpenVZ project with regards to zfs? So far as I'm concerned the license incompatiblity with the zfs/openzfs makes it where it can not be distributed with stuff licensed under the GPL... so I don't really see a way for OpenVZ to ever ship a zfs-enabled kernel... but yeah, if needed they could add support for it in the tools if that makes sense. I'm unclear on what you are looking for other than turning the OpenVZ mailing list into a zfs advocacy group. I do however appreciate you metering wasted disk space by ploop as additional data the OpenVZ devs can work with but as long as ploop isn't using more disk space than the max size of the container disk, I don't really see a problem. While it means one can't over-subscribe the physical disk as much... excessive over-subscription is not ideal either... with wasted space acting as a sort of pre-allocation buffer... and not actually wasted unless the container's disk isn't going to grow in the future. I'd also like to see a comparison between ploop wasted space and that of qcow2... although I'm not sure that qcow2 offers compaction features... since I don't find the word compact in the qemu-img man page. Maybe there is a separate tool for qcow2? TYL, -- Scott Dowdle 704 Church Street Belgrade, MT 59714 (406)388-0827 [home] (406)994-3931 [work] ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
Hello! Because your question is very big I will try to answer in multiple blocks :) --- My disk space issue. 24GB is a wasted space from only one container :) Total wasted space per server is about 900Gb and it's really terrible for me. Why? Because I use server SSD with hardware RAID array's and cost per TB is cosmic! I want to give more fast space to customers instead wasting it! :) --- What I want. What I want from OpenVZ community? I want share my positive experience and build strong community of runners ZFS together with OpenVZ :) Well, I still have one question to openvz team related with behavior of vzctl which is important for ZFS (and another fs too): https://bugzilla.openvz.org/show_bug.cgi?id=3166 --- License issues of ZFS. License issues is not an critical because installing of ZFS is straightforward and do not require any deep integration to system or kernel and work on almost any kernel. And we can call zfs tools (zpool, zfs) without any problems with CDDL license of ZFS. But we can't link to libzfs and fortunately we do not need this. --- Ploop/ext4 vs ZFS ploop builded on top of ext4 and I compare ZFS with ploop and ext4 and many issues notified in my table related with features of both them. Obviously, it's completely incorrect to compare ploop (block level mapper device) with classic filesystem. --- Conclusion Globally, my speech is not related with ZFS itself. It's about storage system for containers. It's most important part of any virtualization technology. Ploop is real revolution in containers world! I really appreciate developers of ploop and love them (and will be happy to bring some beer to they) :) But ploop is not a final step of storage system for containers. And it have big problems described here: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/ploop_issues.md and everybody should know this issues. Ignoring this issues will produce complete data loss on important data! ZFS is not ideal filesystem for containers too! It lacks of very big amount of very important features but it's more reliable and featureful than ploop/ext4 :) Thank you! On Sun, Jan 11, 2015 at 9:57 PM, Scott Dowdle dow...@montanalinux.org wrote: Greetings, - Original Message - And I checked my containers with 200% disk overuse from first message and got negative result. 24Gb of wasted space is not related with cluster size issue. Yeah, but 24GB is a long way off from your original claim (if I remember correctly) of about 900GB... but those probably aren't comparing the same things anyway. I'm lost... because ploop and zfs are not, so far as I can tell, competing filesystems on the same level. zfs is competes with other filesystems like ext4 or xfs... whereas for OpenVZ, so far as I know, there isn't a disk-file-as-disk competitor. Given the popularity and stability of the current qcow2 format popularized by KVM/qemu... and the large number of tools compatible with qcow2 (see libguestfs)... I'm wondering if it would be valuable to add qcow2 support to OpenVZ? You are currently using zfs with OpenVZ, correct? And you didn't have to modify any of the OpenVZ tools in order to do so, correct? If that is the case, what is it you want from the OpenVZ project with regards to zfs? So far as I'm concerned the license incompatiblity with the zfs/openzfs makes it where it can not be distributed with stuff licensed under the GPL... so I don't really see a way for OpenVZ to ever ship a zfs-enabled kernel... but yeah, if needed they could add support for it in the tools if that makes sense. I'm unclear on what you are looking for other than turning the OpenVZ mailing list into a zfs advocacy group. I do however appreciate you metering wasted disk space by ploop as additional data the OpenVZ devs can work with but as long as ploop isn't using more disk space than the max size of the container disk, I don't really see a problem. While it means one can't over-subscribe the physical disk as much... excessive over-subscription is not ideal either... with wasted space acting as a sort of pre-allocation buffer... and not actually wasted unless the container's disk isn't going to grow in the future. I'd also like to see a comparison between ploop wasted space and that of qcow2... although I'm not sure that qcow2 offers compaction features... since I don't find the word compact in the qemu-img man page. Maybe there is a separate tool for qcow2? TYL, -- Scott Dowdle 704 Church Street Belgrade, MT 59714 (406)388-0827 [home] (406)994-3931 [work] ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
BTW, Pavel one issue which you or others might consider and test well before moving to ZFS: 2nd level (i.e. CT user) disk quotas. One will have to emulate Linux quota APIs and quota files for making this work. e.g. some apps like CPanel call quota tools directly and depending on OS installed in container these quota tools expect slightly different Linux quota behavior/APIs. In the past there was a lot of problems with that and we even emulated quota files via /proc. So be warned. On 11 Jan 2015, at 23:16, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello! Because your question is very big I will try to answer in multiple blocks :) --- My disk space issue. 24GB is a wasted space from only one container :) Total wasted space per server is about 900Gb and it's really terrible for me. Why? Because I use server SSD with hardware RAID array's and cost per TB is cosmic! I want to give more fast space to customers instead wasting it! :) --- What I want. What I want from OpenVZ community? I want share my positive experience and build strong community of runners ZFS together with OpenVZ :) Well, I still have one question to openvz team related with behavior of vzctl which is important for ZFS (and another fs too): https://bugzilla.openvz.org/show_bug.cgi?id=3166 --- License issues of ZFS. License issues is not an critical because installing of ZFS is straightforward and do not require any deep integration to system or kernel and work on almost any kernel. And we can call zfs tools (zpool, zfs) without any problems with CDDL license of ZFS. But we can't link to libzfs and fortunately we do not need this. --- Ploop/ext4 vs ZFS ploop builded on top of ext4 and I compare ZFS with ploop and ext4 and many issues notified in my table related with features of both them. Obviously, it's completely incorrect to compare ploop (block level mapper device) with classic filesystem. --- Conclusion Globally, my speech is not related with ZFS itself. It's about storage system for containers. It's most important part of any virtualization technology. Ploop is real revolution in containers world! I really appreciate developers of ploop and love them (and will be happy to bring some beer to they) :) But ploop is not a final step of storage system for containers. And it have big problems described here: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/ploop_issues.md and everybody should know this issues. Ignoring this issues will produce complete data loss on important data! ZFS is not ideal filesystem for containers too! It lacks of very big amount of very important features but it's more reliable and featureful than ploop/ext4 :) Thank you! On Sun, Jan 11, 2015 at 9:57 PM, Scott Dowdle dow...@montanalinux.org wrote: Greetings, - Original Message - And I checked my containers with 200% disk overuse from first message and got negative result. 24Gb of wasted space is not related with cluster size issue. Yeah, but 24GB is a long way off from your original claim (if I remember correctly) of about 900GB... but those probably aren't comparing the same things anyway. I'm lost... because ploop and zfs are not, so far as I can tell, competing filesystems on the same level. zfs is competes with other filesystems like ext4 or xfs... whereas for OpenVZ, so far as I know, there isn't a disk-file-as-disk competitor. Given the popularity and stability of the current qcow2 format popularized by KVM/qemu... and the large number of tools compatible with qcow2 (see libguestfs)... I'm wondering if it would be valuable to add qcow2 support to OpenVZ? You are currently using zfs with OpenVZ, correct? And you didn't have to modify any of the OpenVZ tools in order to do so, correct? If that is the case, what is it you want from the OpenVZ project with regards to zfs? So far as I'm concerned the license incompatiblity with the zfs/openzfs makes it where it can not be distributed with stuff licensed under the GPL... so I don't really see a way for OpenVZ to ever ship a zfs-enabled kernel... but yeah, if needed they could add support for it in the tools if that makes sense. I'm unclear on what you are looking for other than turning the OpenVZ mailing list into a zfs advocacy group. I do however appreciate you metering wasted disk space by ploop as additional data the OpenVZ devs can work with but as long as ploop isn't using more disk space than the max size of the container disk, I don't really see a problem. While it means one can't over-subscribe the physical disk as much... excessive over-subscription is not ideal either... with wasted space acting as a sort of pre-allocation buffer... and not actually wasted unless the container's disk isn't going to grow in the future. I'd also like to see a comparison between ploop wasted space and that of qcow2...
Re: [Users] OpenVZ and ZFS excellent experience
Pavel, it’s impossible to analyze it just by `du` and `df` output, so please give me access if you want me to take a look into it. (e.g. if I would create 10 million of 1KB files du would show me 10GB while ext4 (and most other file systems) would allocate 40GB in reality assuming 4KB block size) Thanks, Kirill On 10 Jan 2015, at 00:54, Pavel Odintsov pavel.odint...@gmail.com wrote: Thank you, Kirill! I am grateful for your answer! I reproduced this issue specially for you on one container with 2.4 times (240% vs 20%) overuse. I do my tests with current vzctl and ploop 1.12.2 (with fixed http://bugzilla.openvz.org/show_bug.cgi?id=3156). Please check this gist: https://gist.github.com/pavel-odintsov/b2162c0f7588bb8e5c15 I can't describe this behavior without complying on ext4 data But I I will be very happy if you fix it :) On Sat, Jan 10, 2015 at 12:29 AM, Kirill Korotaev d...@parallels.com wrote: On 09 Jan 2015, at 21:39, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello, everybody! Do somebody have any news about ZFS and OpenVZ experience? Why not? Did you checked my comparison table for simfs vs ploop vs ZFS volumes? You should do it ASAP: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/openvz_storage_backends.md Still not interesting? For example if you have 5Tb disk array (used up to 90%) and using ploop now you lose about 800GB of disk space! Well, AFAIR we simply have a threshold that ploop is not compacted until it’s size is 20% bigger then it should be… Also you can try smaller ploop block size. Anyway, my point is that it has nothing to do with ext4 metadata as stated in your table. This data is from real HWN with few hundreds of containers. I have excellent experience and very good news about ZFS! ZFS on Linux team will add very important feature, linux quota inside container (more details here https://github.com/zfsonlinux/zfs/pull/2577 But still no news about ZFS from OpenVZ team (and even from Virtuozza Core) and we can work separately :) Fortunately, we do not need any support from vzctl and can use raw vzctl with some lightweight manuals from my repo: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/OpenVZ_containers_on_zfs_filesystem.md I collected all useful information here https://github.com/pavel-odintsov/OpenVZ_ZFS Stay tuned! Join to us! -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
Hello! Thank you! I will contact with you out off list. On Sat, Jan 10, 2015 at 4:44 PM, Kirill Korotaev d...@parallels.com wrote: Pavel, it’s impossible to analyze it just by `du` and `df` output, so please give me access if you want me to take a look into it. (e.g. if I would create 10 million of 1KB files du would show me 10GB while ext4 (and most other file systems) would allocate 40GB in reality assuming 4KB block size) Thanks, Kirill On 10 Jan 2015, at 00:54, Pavel Odintsov pavel.odint...@gmail.com wrote: Thank you, Kirill! I am grateful for your answer! I reproduced this issue specially for you on one container with 2.4 times (240% vs 20%) overuse. I do my tests with current vzctl and ploop 1.12.2 (with fixed http://bugzilla.openvz.org/show_bug.cgi?id=3156). Please check this gist: https://gist.github.com/pavel-odintsov/b2162c0f7588bb8e5c15 I can't describe this behavior without complying on ext4 data But I I will be very happy if you fix it :) On Sat, Jan 10, 2015 at 12:29 AM, Kirill Korotaev d...@parallels.com wrote: On 09 Jan 2015, at 21:39, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello, everybody! Do somebody have any news about ZFS and OpenVZ experience? Why not? Did you checked my comparison table for simfs vs ploop vs ZFS volumes? You should do it ASAP: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/openvz_storage_backends.md Still not interesting? For example if you have 5Tb disk array (used up to 90%) and using ploop now you lose about 800GB of disk space! Well, AFAIR we simply have a threshold that ploop is not compacted until it’s size is 20% bigger then it should be… Also you can try smaller ploop block size. Anyway, my point is that it has nothing to do with ext4 metadata as stated in your table. This data is from real HWN with few hundreds of containers. I have excellent experience and very good news about ZFS! ZFS on Linux team will add very important feature, linux quota inside container (more details here https://github.com/zfsonlinux/zfs/pull/2577 But still no news about ZFS from OpenVZ team (and even from Virtuozza Core) and we can work separately :) Fortunately, we do not need any support from vzctl and can use raw vzctl with some lightweight manuals from my repo: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/OpenVZ_containers_on_zfs_filesystem.md I collected all useful information here https://github.com/pavel-odintsov/OpenVZ_ZFS Stay tuned! Join to us! -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
Thank you, Kirill! I am grateful for your answer! I reproduced this issue specially for you on one container with 2.4 times (240% vs 20%) overuse. I do my tests with current vzctl and ploop 1.12.2 (with fixed http://bugzilla.openvz.org/show_bug.cgi?id=3156). Please check this gist: https://gist.github.com/pavel-odintsov/b2162c0f7588bb8e5c15 I can't describe this behavior without complying on ext4 data But I I will be very happy if you fix it :) On Sat, Jan 10, 2015 at 12:29 AM, Kirill Korotaev d...@parallels.com wrote: On 09 Jan 2015, at 21:39, Pavel Odintsov pavel.odint...@gmail.com wrote: Hello, everybody! Do somebody have any news about ZFS and OpenVZ experience? Why not? Did you checked my comparison table for simfs vs ploop vs ZFS volumes? You should do it ASAP: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/openvz_storage_backends.md Still not interesting? For example if you have 5Tb disk array (used up to 90%) and using ploop now you lose about 800GB of disk space! Well, AFAIR we simply have a threshold that ploop is not compacted until it’s size is 20% bigger then it should be… Also you can try smaller ploop block size. Anyway, my point is that it has nothing to do with ext4 metadata as stated in your table. This data is from real HWN with few hundreds of containers. I have excellent experience and very good news about ZFS! ZFS on Linux team will add very important feature, linux quota inside container (more details here https://github.com/zfsonlinux/zfs/pull/2577 But still no news about ZFS from OpenVZ team (and even from Virtuozza Core) and we can work separately :) Fortunately, we do not need any support from vzctl and can use raw vzctl with some lightweight manuals from my repo: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/OpenVZ_containers_on_zfs_filesystem.md I collected all useful information here https://github.com/pavel-odintsov/OpenVZ_ZFS Stay tuned! Join to us! -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
Re: [Users] OpenVZ and ZFS excellent experience
It is also important to note that there is wasted space with ZFS as is right now if you use advanced format drives (usually 2TB or larger). When using ashift=12 (4k sector size) to create a ZFS raid, you'll lose about 10-20% of your disk capacity with ZFS depending on the RAID type, I don't remember if this affects stripes or not. It is most noticeable in my testing on a RAIDZ2. When using ashift=9 (512 sector size), you'll have the full capacity but performance will suffer on advanced format drives. I've also opened up a performance bug about the write performance and amount of data written to the devices far exceeding the initial write size which seems to be pretty noticeable when your writes don't align perfectly with the zfs block size. I had an email with you about this. However, there are a lot of useful features in ZFS that may, or may not, be worth this capacity loss and write performance limitations depending on your use case. I have hopes the performance issues are being worked out with the upcoming releases. Pavel Odintsov mailto:pavel.odint...@gmail.com Friday, January 9, 2015 3:39 PM Hello, everybody! Do somebody have any news about ZFS and OpenVZ experience? Why not? Did you checked my comparison table for simfs vs ploop vs ZFS volumes? You should do it ASAP: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/openvz_storage_backends.md Still not interesting? For example if you have 5Tb disk array (used up to 90%) and using ploop now you lose about 800GB of disk space! This data is from real HWN with few hundreds of containers. I have excellent experience and very good news about ZFS! ZFS on Linux team will add very important feature, linux quota inside container (more details here https://github.com/zfsonlinux/zfs/pull/2577 But still no news about ZFS from OpenVZ team (and even from Virtuozza Core) and we can work separately :) Fortunately, we do not need any support from vzctl and can use raw vzctl with some lightweight manuals from my repo: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/OpenVZ_containers_on_zfs_filesystem.md I collected all useful information here https://github.com/pavel-odintsov/OpenVZ_ZFS Stay tuned! Join to us! ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
[Users] OpenVZ and ZFS excellent experience
Hello, everybody! Do somebody have any news about ZFS and OpenVZ experience? Why not? Did you checked my comparison table for simfs vs ploop vs ZFS volumes? You should do it ASAP: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/openvz_storage_backends.md Still not interesting? For example if you have 5Tb disk array (used up to 90%) and using ploop now you lose about 800GB of disk space! This data is from real HWN with few hundreds of containers. I have excellent experience and very good news about ZFS! ZFS on Linux team will add very important feature, linux quota inside container (more details here https://github.com/zfsonlinux/zfs/pull/2577 But still no news about ZFS from OpenVZ team (and even from Virtuozza Core) and we can work separately :) Fortunately, we do not need any support from vzctl and can use raw vzctl with some lightweight manuals from my repo: https://github.com/pavel-odintsov/OpenVZ_ZFS/blob/master/OpenVZ_containers_on_zfs_filesystem.md I collected all useful information here https://github.com/pavel-odintsov/OpenVZ_ZFS Stay tuned! Join to us! -- Sincerely yours, Pavel Odintsov ___ Users mailing list Users@openvz.org https://lists.openvz.org/mailman/listinfo/users
