Re: [openstack-dev] Swift: reason for using xfs on devices
On Wednesday, July 02, 2014 1:06 PM, Pete Zaitcev zait...@redhat.com wrote: Thanks for the detailed explanation. Let me clarify the behavior of swift. (1) Use ext4 on devices. (2) Corrupt the data on (1)'s filesystem (3) Move corrupt files to lost+found without a trace by ext4's fsck (4) Cannot recognize (3) by Swift's auditors so hashes.pkl is not updated. Is above sequence correct? If it's correct, I understand we better to use xfs. Thanks in advance, Hisashi Osanai -Original Message- From: Pete Zaitcev [mailto:zait...@redhat.com] Sent: Wednesday, July 02, 2014 1:06 PM To: Osanai, Hisashi/小山内 尚 Cc: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Swift: reason for using xfs on devices On Wed, 2 Jul 2014 00:16:42 + Osanai, Hisashi osanai.hisa...@jp.fujitsu.com wrote: So I think if performance of swift is more important rather than scalability of it, it is a good idea to use ext4. The real problem is what happens when your drives corrupt the data. Both ext4 and XFS demonstrated good resilience, but XFS leaves empty files in directories where corrupt files were, while ext4's fsck moves them to lost+found without a trace. When that happens, Swift's auditors cannot know that something was amiss and the replication is not triggered (because hash lists are only updated by auditors). Mr. You Yamagata worked on a patch to address this problem, but did not complete it. See here: https://review.openstack.org/11452 -- Pete ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift: reason for using xfs on devices
On Tue, Jul 1, 2014 at 6:21 AM, Osanai, Hisashi osanai.hisa...@jp.fujitsu.com wrote: Hi, In the following document, there is a setup up procedure for storage and it seems that swift recommends to use xfs. http://docs.openstack.org/icehouse/install-guide/install/yum/content/installing-and-configuring-storage-nodes.html === 2. For each device on the node that you want to use for storage, set up the XFS volume (/dev/sdb is used as an example). Use a single partition per drive. For example, in a server with 12 disks you may use one or two disks for the operating system which should not be touched in this step. The other 10 or 11 disks should be partitioned with a single partition, then formatted in XFS. === I would like to know the reason why swift recommends xfs rather than ext4? The install guide only recommends a single path, not many options, to ensure success. There's a little bit of discussion in the developer docs: http://docs.openstack.org/developer/swift/deployment_guide.html#filesystem-considerations I think that packstack gives the option of using xfs or ext4, so there must be sufficient testing for ext4. Anne I think ext4 has reasonable performance and can support 1EiB from design point of view. # The max file system size of ext4 is not enough??? Thanks in advance, Hisashi Osanai ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift: reason for using xfs on devices
On Tuesday, July 01, 2014 9:44 PM, Anne Gentle a...@openstack.org wrote: Thank you for the quick response. The install guide only recommends a single path, not many options, to ensure success. I understand the point for writing the document. There's a little bit of discussion in the developer docs: http://docs.openstack.org/developer/swift/deployment_guide.html#filesystem-considerations I think that packstack gives the option of using xfs or ext4, so there must be sufficient testing for ext4. Thank you for this info. In the discussion, there is a following sentence. After thorough testing with our use cases and hardware configurations, XFS was the best all-around choice. I would like to know what kind of testing should I do from filesystem point of view? Backgroup of this question is: I read the following performance comparison about ext4 and xfs. There are some results of Benchmark. It seems that performance of ext4 is better than xfs (Eric Whitney's FFSB testing). So I think if performance of swift is more important rather than scalability of it, it is a good idea to use ext4. http://www.linuxtag.org/2013/fileadmin/www.linuxtag.org/slides/Heinz_Mauelshagen_-_Which_filesystem_should_I_use_.e204.pdf Best Regards, Hisashi Osanai From: Anne Gentle [mailto:a...@openstack.org] Sent: Tuesday, July 01, 2014 9:44 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Swift: reason for using xfs on devices On Tue, Jul 1, 2014 at 6:21 AM, Osanai, Hisashi osanai.hisa...@jp.fujitsu.com wrote: Hi, In the following document, there is a setup up procedure for storage and it seems that swift recommends to use xfs. http://docs.openstack.org/icehouse/install-guide/install/yum/content/installing-and-configuring-storage-nodes.html === 2. For each device on the node that you want to use for storage, set up the XFS volume (/dev/sdb is used as an example). Use a single partition per drive. For example, in a server with 12 disks you may use one or two disks for the operating system which should not be touched in this step. The other 10 or 11 disks should be partitioned with a single partition, then formatted in XFS. === I would like to know the reason why swift recommends xfs rather than ext4? The install guide only recommends a single path, not many options, to ensure success. There's a little bit of discussion in the developer docs: http://docs.openstack.org/developer/swift/deployment_guide.html#filesystem-considerations I think that packstack gives the option of using xfs or ext4, so there must be sufficient testing for ext4. Anne I think ext4 has reasonable performance and can support 1EiB from design point of view. # The max file system size of ext4 is not enough??? Thanks in advance, Hisashi Osanai ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift: reason for using xfs on devices
On Wed, 2 Jul 2014 00:16:42 + Osanai, Hisashi osanai.hisa...@jp.fujitsu.com wrote: So I think if performance of swift is more important rather than scalability of it, it is a good idea to use ext4. The real problem is what happens when your drives corrupt the data. Both ext4 and XFS demonstrated good resilience, but XFS leaves empty files in directories where corrupt files were, while ext4's fsck moves them to lost+found without a trace. When that happens, Swift's auditors cannot know that something was amiss and the replication is not triggered (because hash lists are only updated by auditors). Mr. You Yamagata worked on a patch to address this problem, but did not complete it. See here: https://review.openstack.org/11452 -- Pete ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
