nagraham opened a new issue, #6252: URL: https://github.com/apache/opendal/issues/6252
### Context I'm working on writing data to R2 Data Catalog (https://developers.cloudflare.com/r2/data-catalog/) with iceberg-rust, which uses OpenDAL. ### Problem When writing large files into the Iceberg table (on R2), I get the following error: ``` S3Error { code: "InvalidPart", message: "All non-trailing parts must have the same length.", resource: "", request_id: "" } ``` I traced it to the Multi-part upload in OpenDAL. It seems this was fixed a while back (https://github.com/apache/opendal/issues/2904), which a change which introduced a `enable_exact_buf_write` configuration value, which is still referenced [in the docs under Cloudflare R2](https://opendal.apache.org/docs/rust/opendal/services/struct.S3.html#cloudflare-r2). However, that config value doesn't exist anymore. I guess it was deprecated in v0.42 ([change log](https://github.com/apache/opendal/blob/de6f5e4f7fa8da846995d4bfb6c0e3d183732fbf/core/src/docs/upgrade.md?plain=1#L541)), and replaced with `BufferedWriter`. But multi-part uploads are failing with that issue (see some code below which can reproduce the error). Perhaps there has been a regression to the BufferedWriter? I notice the R2 Integ tests are disabled (https://github.com/apache/opendal/issues/4324), so perhaps a regression snuck in? ``` # This is using opendal = { version = "0.53.0", features = ["services-s3"] } let mut cfg: S3Config = S3Config::default(); // My configuration let operator = Operator::new(builder).unwrap().finish(); let mut w = operator.writer(&cli.object).await.expect("A writer"); // Does OpenDAL write to a buffer and then write the same sized chunks in a multipart upload? // Write 10MB w.write(vec![0; 10 * 1024 * 1024]) .await .map_err(|e| format!("Error writing first 10MB: {e:?}"))?; // Write 20MB w.write(vec![0; 20 * 1024 * 1024]) .await .map_err(|e| format!("Error writing second 20MB chunk: {e:?}"))?; // Write 8MB w.write(vec![0; 8 * 1024 * 1024]) .await .map_err(|e| format!("Error writing third 8MB chunk: {e:?}"))?; w.close() .await .map_err(|e| format!("Error while closing the writer: {e:?}"))?; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@opendal.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
