----- Original Message ----- > From: "Abhishek Dixit" <[email protected]> > To: "ceph-devel" <[email protected]> > Sent: Thursday, June 11, 2015 8:28:14 AM > Subject: Multipart Upload : Limit on no of parts > > Hi, > > I was looking into limit imposed on no of uploads for multipart upload. > The way it works currently seems as below: > 1. RGWCompleteMultipart::get_params does following > ______________________________________________ > #define COMPLETE_MULTIPART_MAX_LEN (1024 * 1024) /* api defines max > 10,000 parts, this should be enough */ > ret = rgw_rest_read_all_input(s, &data, &len, COMPLETE_MULTIPART_MAX_LEN); > ___________________________________________________________ > > 2. rgw_rest_read_all_input compares cl with max_len where > cl is Content Length value from POST multipart complete request header and > max_len is COMPLETE_MULTIPART_MAX_LEN as defined above. > > 3. For chunked input, again the length read is compared with > COMPLETE_MULTIPART_MAX_LEN. > > For 10104 parts POST request, we had cl:1039762 and max_len:1048576 > which worked. > Limit on no of parts seems to be imposed by this mechanism rather than > an explicit parts check. > > Kindly let me know if the understanding is correct. > In case it is, should we not impose a limit via explicit checking for > total no of parts? > Also, can we make this parts limit a configurable option via config_opts? >
I thought we had an explicit check for that, but I can't really find it, so I might have gotten it wrong. We should enforce some limit on that number. First, the way we index the parts in the omap assumes a certain max so it pads the keys with a specific amount of zeros. Second, in the case where users are uploading parts with different sizes, the created manifest will be too large. Is there a real need to have more than 10k parts? Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
