Re: [fuse-devel] [RFC PATCH 0/5] fuse: make maximum read/write request size tunable
Hi Miklos, Thank you for your comments. (2012/07/12 19:13), Miklos Szeredi wrote: HAYASAKA Mitsuo writes: Hi Yuan and Han-Wen, Thank you for your comments. (2012/07/06 22:58), Han-Wen Nienhuys wrote: On Fri, Jul 6, 2012 at 2:53 AM, Liu Yuan wrote: On 07/05/2012 06:50 PM, Mitsuo Hayasaka wrote: One of the ways to solve this is to make them tunable. In this series, the new sysfs parameter max_pages_per_req is introduced. It limits the maximum read/write size in fuse request and it can be changed from 32 to 256 pages in current implementations. When the max_read/max_write mount option is specified, FUSE request size is set per mount. (The size is rounded-up to page size and limited up to max_pages_per_req.) Why maxim 256 pages? If we are here, we can go further: most of object storage system has object size of multiple to dozens of megabytes. So I think probably 1M is too small. Our distribution storage system has 4M per object, so I think at least maxim size could be bigger than 4M. The maximum pipe size on my system is 1M, so if you go beyond that, splicing from the FD won't work. Also, the userspace client must reserve a buffer this size so it can receive a write, which is a waste since most requests are much smaller. I checked the maximum pipe size can be changed using fcntl(2) or /proc/sys/fs/pipe-max-size. It is clear that it is not a fixed value. Also, it seems that there is a request for setting the maximum number of pages per fuse request to 4M (1024 pages). One of the reasons to introduce the sysfs max_pages_per_req parameter is to set a threshold of the maximum number of pages dynamically according to the administrator's demand, and root can only change it. So, when the maximum value is required to be set to not more than the pipe-max-size, the max_pages_per_req should be changed considering it. It seems that the upper limit of this parameter does not have to be not more than it. I'm planning to limit max_pages_per_req up to 1024 pages and add the document to /Documentation/filesystems/fuse.txt, as follows. "the sysfs max_pages_per_req parameter can be changed from 32 to 1024. The default is 32 pages. Generally, the pipe-max-size is 1M (256 pages) and it is better to set it to not more than the pipe-max-size." Can't we just use pipe-max-size for the limit? This is great! I'd like to change this patch to using the pipe-max-size for the upper limit of the max_pages_per_req sysfs paramter, and resubmit it. Then we'll use the minimum of pipe-max-size and max_read/max_write for sizing the requests. Another comment: do we really need to allocate each and every request with space for the pages? I don't think that makes sense. Let's leave some small number of pages inline in the request and allocate a separate array if the number of pages is too large. There may even be some utilities in the kernel to handle dynamically sized page arrays (I haven't looked but I suspect there is). This is interesting and enables to dramatically reduce the number of page allocation and free. However, it seems that it is necessary to investigate if this is feasible. Thanks, Thanks, Miklos -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [fuse-devel] [RFC PATCH 0/5] fuse: make maximum read/write request size tunable
Hi Miklos, Thank you for your comments. (2012/07/12 19:13), Miklos Szeredi wrote: HAYASAKA Mitsuomitsuo.hayasaka...@hitachi.com writes: Hi Yuan and Han-Wen, Thank you for your comments. (2012/07/06 22:58), Han-Wen Nienhuys wrote: On Fri, Jul 6, 2012 at 2:53 AM, Liu Yuannamei.u...@gmail.com wrote: On 07/05/2012 06:50 PM, Mitsuo Hayasaka wrote: One of the ways to solve this is to make them tunable. In this series, the new sysfs parameter max_pages_per_req is introduced. It limits the maximum read/write size in fuse request and it can be changed from 32 to 256 pages in current implementations. When the max_read/max_write mount option is specified, FUSE request size is set per mount. (The size is rounded-up to page size and limited up to max_pages_per_req.) Why maxim 256 pages? If we are here, we can go further: most of object storage system has object size of multiple to dozens of megabytes. So I think probably 1M is too small. Our distribution storage system has 4M per object, so I think at least maxim size could be bigger than 4M. The maximum pipe size on my system is 1M, so if you go beyond that, splicing from the FD won't work. Also, the userspace client must reserve a buffer this size so it can receive a write, which is a waste since most requests are much smaller. I checked the maximum pipe size can be changed using fcntl(2) or /proc/sys/fs/pipe-max-size. It is clear that it is not a fixed value. Also, it seems that there is a request for setting the maximum number of pages per fuse request to 4M (1024 pages). One of the reasons to introduce the sysfs max_pages_per_req parameter is to set a threshold of the maximum number of pages dynamically according to the administrator's demand, and root can only change it. So, when the maximum value is required to be set to not more than the pipe-max-size, the max_pages_per_req should be changed considering it. It seems that the upper limit of this parameter does not have to be not more than it. I'm planning to limit max_pages_per_req up to 1024 pages and add the document to /Documentation/filesystems/fuse.txt, as follows. the sysfs max_pages_per_req parameter can be changed from 32 to 1024. The default is 32 pages. Generally, the pipe-max-size is 1M (256 pages) and it is better to set it to not more than the pipe-max-size. Can't we just use pipe-max-size for the limit? This is great! I'd like to change this patch to using the pipe-max-size for the upper limit of the max_pages_per_req sysfs paramter, and resubmit it. Then we'll use the minimum of pipe-max-size and max_read/max_write for sizing the requests. Another comment: do we really need to allocate each and every request with space for the pages? I don't think that makes sense. Let's leave some small number of pages inline in the request and allocate a separate array if the number of pages is too large. There may even be some utilities in the kernel to handle dynamically sized page arrays (I haven't looked but I suspect there is). This is interesting and enables to dramatically reduce the number of page allocation and free. However, it seems that it is necessary to investigate if this is feasible. Thanks, Thanks, Miklos -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 5/5] fuse: add documentation of sysfs parameter to limit maximum fuse request size
Hi Rob, Thank you for your comments. (2012/07/06 21:54), Rob Landley wrote: On 07/05/2012 05:51 AM, Mitsuo Hayasaka wrote: Add an explantion about the sysfs parameter to the limit maximum read/write request size. Signed-off-by: Mitsuo Hayasaka Cc: Rob Landley Cc: Miklos Szeredi --- Documentation/filesystems/fuse.txt | 17 - 1 files changed, 16 insertions(+), 1 deletions(-) diff --git a/Documentation/filesystems/fuse.txt b/Documentation/filesystems/fuse.txt index 13af4a4..e6ffba3 100644 --- a/Documentation/filesystems/fuse.txt +++ b/Documentation/filesystems/fuse.txt @@ -108,13 +108,28 @@ Mount options With this option the maximum size of read operations can be set. The default is infinite. Note that the size of read requests is - limited anyway to 32 pages (which is 128kbyte on i386). + limited to 32 pages (which is 128kbyte on i386) if direct_io + option is not specified. When direct_io option is specified, + the request size is limited to max_pages_per_req sysfs parameter. "Note that the maximum size of read requests defaults to 32 pages (128k on i386), use max_pages_per_req to change this default." And then describe max_page_per_req sufficiently thoroughly below, all in one place. OK, I will revise it. (By the way, has anybody actually tested it with a single page as the limit? Does that work?) This patch series enables the maximum request size to change to arbitrary number from 32 to 256, and cannot set it to less than 32 pages. 'blksize=N' Set the block size for the filesystem. The default is 512. This option is only valid for 'fuseblk' type mounts. +Sysfs parameter +~~~ + +The sysfs parameter max_pages_per_req limits the maximum page size per +FUSE request. No, it limits the maximum size of a data request and the units are decimal number of pages. It doesn't change the size of memory pages in the system. You are right. I will revise it. Also, your first hunk implies this setting only takes effect if they mounted with "-o direct_io", is that true? The request size increases using max_pages_per_req for read operation w/ direct_io, and write operation w/ and w/o direct_io. But it is not changed for read operation w/o direct_io. So, it is true if only focusing on read operation. + /sys/fs/fuse/max_pages_per_req + +The default is 32 pages. It can be changed from 32 to 256 pages, which +may improve the read/write throughput optimizing it. This change is +effective per mount. Therefore, the re-mounting of FUSE filesystem +is required after changing it. I'd say "Changing it to 256 pages may improve read/write throguhput on systems with enough memory. Existing FUSE mounts must be remounted for this change to take effect." I.E. don't imply 32 and 256 are the only options unless they are. (Is there some requirement that it be a power of 2, or just a good idea?) Here, I wanted to imply that the max_paegs_per_req can be changed to arbitrary number from 32 to 256. I will revise it since this explanation is misleading. Also, there is no requirement that it be a power of 2 although it is a good idea if only focusing on kmalloc(). One of the reasons to introduce the max_pages_per_req sysfs parameter is to let the libfuse get the current maximum request size and change the MIN_BUFSIZE limitation according to it to avoid an waste of memory in userspace. And per-mount sounds like you're setting it for a specific mount point, so if I have three mounts there would be three entries under /sys/fs/fuse, which does not seem to be the case. (Which is odd, because you'd think there would be an "-o max_pages_per_req=128" that _would_ set this per-mount if the value actually used is cached in the superblock, but I'm not seeing one...) The max_pages_per_req is a system limitation controlled by the administrator. The actual number of allocated pages per request can be changed using max_read/max_write mount options below this system limitation. I will revise and resubmit this patch series soon. Thanks, Rob -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 5/5] fuse: add documentation of sysfs parameter to limit maximum fuse request size
Hi Rob, Thank you for your comments. (2012/07/06 21:54), Rob Landley wrote: On 07/05/2012 05:51 AM, Mitsuo Hayasaka wrote: Add an explantion about the sysfs parameter to the limit maximum read/write request size. Signed-off-by: Mitsuo Hayasakamitsuo.hayasaka...@hitachi.com Cc: Rob Landleyr...@landley.net Cc: Miklos Szeredimik...@szeredi.hu --- Documentation/filesystems/fuse.txt | 17 - 1 files changed, 16 insertions(+), 1 deletions(-) diff --git a/Documentation/filesystems/fuse.txt b/Documentation/filesystems/fuse.txt index 13af4a4..e6ffba3 100644 --- a/Documentation/filesystems/fuse.txt +++ b/Documentation/filesystems/fuse.txt @@ -108,13 +108,28 @@ Mount options With this option the maximum size of read operations can be set. The default is infinite. Note that the size of read requests is - limited anyway to 32 pages (which is 128kbyte on i386). + limited to 32 pages (which is 128kbyte on i386) if direct_io + option is not specified. When direct_io option is specified, + the request size is limited to max_pages_per_req sysfs parameter. Note that the maximum size of read requests defaults to 32 pages (128k on i386), use max_pages_per_req to change this default. And then describe max_page_per_req sufficiently thoroughly below, all in one place. OK, I will revise it. (By the way, has anybody actually tested it with a single page as the limit? Does that work?) This patch series enables the maximum request size to change to arbitrary number from 32 to 256, and cannot set it to less than 32 pages. 'blksize=N' Set the block size for the filesystem. The default is 512. This option is only valid for 'fuseblk' type mounts. +Sysfs parameter +~~~ + +The sysfs parameter max_pages_per_req limits the maximum page size per +FUSE request. No, it limits the maximum size of a data request and the units are decimal number of pages. It doesn't change the size of memory pages in the system. You are right. I will revise it. Also, your first hunk implies this setting only takes effect if they mounted with -o direct_io, is that true? The request size increases using max_pages_per_req for read operation w/ direct_io, and write operation w/ and w/o direct_io. But it is not changed for read operation w/o direct_io. So, it is true if only focusing on read operation. + /sys/fs/fuse/max_pages_per_req + +The default is 32 pages. It can be changed from 32 to 256 pages, which +may improve the read/write throughput optimizing it. This change is +effective per mount. Therefore, the re-mounting of FUSE filesystem +is required after changing it. I'd say Changing it to 256 pages may improve read/write throguhput on systems with enough memory. Existing FUSE mounts must be remounted for this change to take effect. I.E. don't imply 32 and 256 are the only options unless they are. (Is there some requirement that it be a power of 2, or just a good idea?) Here, I wanted to imply that the max_paegs_per_req can be changed to arbitrary number from 32 to 256. I will revise it since this explanation is misleading. Also, there is no requirement that it be a power of 2 although it is a good idea if only focusing on kmalloc(). One of the reasons to introduce the max_pages_per_req sysfs parameter is to let the libfuse get the current maximum request size and change the MIN_BUFSIZE limitation according to it to avoid an waste of memory in userspace. And per-mount sounds like you're setting it for a specific mount point, so if I have three mounts there would be three entries under /sys/fs/fuse, which does not seem to be the case. (Which is odd, because you'd think there would be an -o max_pages_per_req=128 that _would_ set this per-mount if the value actually used is cached in the superblock, but I'm not seeing one...) The max_pages_per_req is a system limitation controlled by the administrator. The actual number of allocated pages per request can be changed using max_read/max_write mount options below this system limitation. I will revise and resubmit this patch series soon. Thanks, Rob -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [fuse-devel] [RFC PATCH 0/5] fuse: make maximum read/write request size tunable
Hi Yuan and Han-Wen, Thank you for your comments. (2012/07/06 22:58), Han-Wen Nienhuys wrote: On Fri, Jul 6, 2012 at 2:53 AM, Liu Yuan wrote: On 07/05/2012 06:50 PM, Mitsuo Hayasaka wrote: One of the ways to solve this is to make them tunable. In this series, the new sysfs parameter max_pages_per_req is introduced. It limits the maximum read/write size in fuse request and it can be changed from 32 to 256 pages in current implementations. When the max_read/max_write mount option is specified, FUSE request size is set per mount. (The size is rounded-up to page size and limited up to max_pages_per_req.) Why maxim 256 pages? If we are here, we can go further: most of object storage system has object size of multiple to dozens of megabytes. So I think probably 1M is too small. Our distribution storage system has 4M per object, so I think at least maxim size could be bigger than 4M. The maximum pipe size on my system is 1M, so if you go beyond that, splicing from the FD won't work. Also, the userspace client must reserve a buffer this size so it can receive a write, which is a waste since most requests are much smaller. I checked the maximum pipe size can be changed using fcntl(2) or /proc/sys/fs/pipe-max-size. It is clear that it is not a fixed value. Also, it seems that there is a request for setting the maximum number of pages per fuse request to 4M (1024 pages). One of the reasons to introduce the sysfs max_pages_per_req parameter is to set a threshold of the maximum number of pages dynamically according to the administrator's demand, and root can only change it. So, when the maximum value is required to be set to not more than the pipe-max-size, the max_pages_per_req should be changed considering it. It seems that the upper limit of this parameter does not have to be not more than it. I'm planning to limit max_pages_per_req up to 1024 pages and add the document to /Documentation/filesystems/fuse.txt, as follows. "the sysfs max_pages_per_req parameter can be changed from 32 to 1024. The default is 32 pages. Generally, the pipe-max-size is 1M (256 pages) and it is better to set it to not more than the pipe-max-size." This is just a plan and any comments are appreciated. Thanks, -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [fuse-devel] [RFC PATCH 0/5] fuse: make maximum read/write request size tunable
Hi Yuan and Han-Wen, Thank you for your comments. (2012/07/06 22:58), Han-Wen Nienhuys wrote: On Fri, Jul 6, 2012 at 2:53 AM, Liu Yuannamei.u...@gmail.com wrote: On 07/05/2012 06:50 PM, Mitsuo Hayasaka wrote: One of the ways to solve this is to make them tunable. In this series, the new sysfs parameter max_pages_per_req is introduced. It limits the maximum read/write size in fuse request and it can be changed from 32 to 256 pages in current implementations. When the max_read/max_write mount option is specified, FUSE request size is set per mount. (The size is rounded-up to page size and limited up to max_pages_per_req.) Why maxim 256 pages? If we are here, we can go further: most of object storage system has object size of multiple to dozens of megabytes. So I think probably 1M is too small. Our distribution storage system has 4M per object, so I think at least maxim size could be bigger than 4M. The maximum pipe size on my system is 1M, so if you go beyond that, splicing from the FD won't work. Also, the userspace client must reserve a buffer this size so it can receive a write, which is a waste since most requests are much smaller. I checked the maximum pipe size can be changed using fcntl(2) or /proc/sys/fs/pipe-max-size. It is clear that it is not a fixed value. Also, it seems that there is a request for setting the maximum number of pages per fuse request to 4M (1024 pages). One of the reasons to introduce the sysfs max_pages_per_req parameter is to set a threshold of the maximum number of pages dynamically according to the administrator's demand, and root can only change it. So, when the maximum value is required to be set to not more than the pipe-max-size, the max_pages_per_req should be changed considering it. It seems that the upper limit of this parameter does not have to be not more than it. I'm planning to limit max_pages_per_req up to 1024 pages and add the document to /Documentation/filesystems/fuse.txt, as follows. the sysfs max_pages_per_req parameter can be changed from 32 to 1024. The default is 32 pages. Generally, the pipe-max-size is 1M (256 pages) and it is better to set it to not more than the pipe-max-size. This is just a plan and any comments are appreciated. Thanks, -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
