lhotari opened a new issue, #15159: URL: https://github.com/apache/pulsar/issues/15159
**Describe the bug** Offloading large ledgers fail with Google Cloud Storage and the default settings. With default settings offloading ledgers over 2GB will fail to GCS since there's a limitation in JClouds (reported as [[JCLOUDS-1606] Cannot upload more than 32 parts to GCS](https://issues.apache.org/jira/browse/JCLOUDS-1606)) which limits a multipart upload to 32 parts. GCS supports multipart uploads up to 10000 parts, but JClouds doesn't use the API in a way to achieve more than 32 parts. Here's an example log entry of the problem: ``` java.util.concurrent.CompletionException: org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{ "error": { "code": 400, "message": "The number of source components provided (35) exceeds the maximum (32)", "errors": [ { "message": "The number of source components provided (35) exceeds the maximum (32)", "domain": "global", "reason": "invalid" } ] } } ] at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?] at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?] at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?] at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$offload$0(BlobStoreManagedLedgerOffloader.java:237) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) [com.google.guava-guava-31.0.1-jre.jar:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?] at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$offload$0(BlobStoreManagedLedgerOffloader.java:237) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) [com.google.guava-guava-31.0.1-jre.jar:?] at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) [com.google.guava-guava-31.0.1-jre.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) [com.google.guava-guava-31.0.1-jre.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{ "error": { "code": 400, "message": "The number of source components provided (35) exceeds the maximum (32)", "errors": [ { "message": "The number of source components provided (35) exceeds the maximum (32)", "domain": "global", "reason": "invalid" } ] } } ] at org.jclouds.googlecloudstorage.handlers.GoogleCloudStorageErrorHandler.handleError(GoogleCloudStorageErrorHandler.java:40) ~[?:?] at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[?:?] at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138) ~[?:?] at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107) ~[?:?] at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91) ~[?:?] at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74) ~[?:?] at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45) ~[?:?] at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[?:?] at org.apache.pulsar.jcloud.shade.com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:89) ~[?:?] at com.sun.proxy.$Proxy144.composeObjects(Unknown Source) ~[?:?] at org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.completeMultipartUpload(GoogleCloudStorageBlobStore.java:405) ~[?:?] at jdk.internal.reflect.GeneratedMethodAccessor373.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.apache.pulsar.jcloud.shade.com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) ~[?:?] at com.sun.proxy.$Proxy88.completeMultipartUpload(Unknown Source) ~[?:?] at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$offload$0(BlobStoreManagedLedgerOffloader.java:226) ~[?:?] ... 11 more ``` **To Reproduce** Steps to reproduce the behavior: 1. Configure offloading to use GCS 2. Produce a lot of messages 3. Trigger ledger offloading **Expected behavior** Ledger offloading should work with large ledger sizes **Workaround** ``` # ensure that ledgers don't grow over 1500MB managedLedgerMaxSizePerLedgerMbytes=1500 managedLedgerMinLedgerRolloverTimeMinutes=0 ``` or ``` # increase block size to 128MB, maximum file size doubles to 4GB gcsManagedLedgerOffloadMaxBlockSizeInBytes=134217728 ``` **Additional context** JClouds issue reported as [[JCLOUDS-1606] Cannot upload more than 32 parts to GCS](https://issues.apache.org/jira/browse/JCLOUDS-1606). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
