lhotari commented on PR #22220: URL: https://github.com/apache/pulsar/pull/22220#issuecomment-2069324241
> @lhotari With the above fix I tried testing GCS offloader with ledgers of size >= 2.5gb. I'm still facing the same error > > ``` > Error in offload > null > > Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/bucket-cognitree-ls1258/o/d7f69abd-dc45-4fc1-bbe3-bd9daaa8fea7-ledger-9/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{ > "error": { > "code": 400, > "message": "The number of source components provided (38) exceeds the maximum (32)", > "errors": [ > { > "message": "The number of source components provided (38) exceeds the maximum (32)", > "domain": "global", > "reason": "invalid" > } > ] > } > } > ] > ``` > > I have also written a simple code using Jclouds 2.6.0 to upload a large file of size 3gb to GCS bucket, it has uploaded successfully. @nikhil-ctds Do you have a chance to rerun the test with `gcsManagedLedgerOffloadMaxBlockSizeInBytes=134217728` ? The default value `gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864` will limit the max upload size to 2048MiB (64MiB * 32). Doubling gcsManagedLedgerOffloadMaxBlockSizeInBytes to `134217728` should lift the limit to 4096MiB. Can you please verify? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
