Re: [ceph-users] RadosGW Direct Upload Limitation
- Original Message - From: Craig Lewis cle...@centraldesktop.com To: Gregory Farnum g...@gregs42.com Cc: ceph-users@lists.ceph.com Sent: Monday, March 16, 2015 11:48:15 AM Subject: Re: [ceph-users] RadosGW Direct Upload Limitation Maybe, but I'm not sure if Yehuda would want to take it upstream or not. This limit is present because it's part of the S3 spec. For larger objects you should use multi-part upload, which can get much bigger. -Greg Note that the multi-part upload has a lower limit of 4MiB per part, and the direct upload has an upper limit of 5GiB. The limit is 10MB, but it does not apply to the last part, so basically you could upload any object size with it. I would still recommend using the plain upload for smaller object sizes, it is faster, and the resulting object might be more efficient (for really small sizes). Yehuda So you have to use both methods - direct upload for small files, and multi-part upload for big files. Your best bet is to use the Amazon S3 libraries. They have functions that take care of it for you. I'd like to see this mentioned in the Ceph documentation someplace. When I first encountered the issue, I couldn't find a limit in the RadosGW documentation anywhere. I only found the 5GiB limit in the Amazon API documentation, which lead me to test on RadosGW. Now that I know it was done to preserve Amazon compatibility, I don't want to override the value anymore. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RadosGW Direct Upload Limitation
Maybe, but I'm not sure if Yehuda would want to take it upstream or not. This limit is present because it's part of the S3 spec. For larger objects you should use multi-part upload, which can get much bigger. -Greg Note that the multi-part upload has a lower limit of 4MiB per part, and the direct upload has an upper limit of 5GiB. So you have to use both methods - direct upload for small files, and multi-part upload for big files. Your best bet is to use the Amazon S3 libraries. They have functions that take care of it for you. I'd like to see this mentioned in the Ceph documentation someplace. When I first encountered the issue, I couldn't find a limit in the RadosGW documentation anywhere. I only found the 5GiB limit in the Amazon API documentation, which lead me to test on RadosGW. Now that I know it was done to preserve Amazon compatibility, I don't want to override the value anymore. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RadosGW Direct Upload Limitation
On Mon, Mar 16, 2015 at 11:14 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Hi all! I have recently updated to CEPH version 0.80.9 (latest Firefly release) which presumably supports direct upload. I 've tried to upload a file using this functionality and it seems that is working for files up to 5GB. For files above 5GB there is an error. I believe that this is because of a hardcoded limit: #define RGW_MAX_PUT_SIZE(5ULL*1024*1024*1024) Is there a way to increase that limit other than compiling CEPH from source? No. Could we somehow put it as a configuration parameter? Maybe, but I'm not sure if Yehuda would want to take it upstream or not. This limit is present because it's part of the S3 spec. For larger objects you should use multi-part upload, which can get much bigger. -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
