dionjansen commented on issue #3833:
URL: https://github.com/apache/pulsar/issues/3833#issuecomment-644844173
@wolfstudy I'm getting the same with pulsar `2.5.0` also I use a
non-standard region `eu-west-1`
My pulsar config:
```yaml
managedLedgerOffloadDriver: aws-s3
s3ManagedLedgerOffloadBucket: test-offload-bucket
s3ManagedLedgerOffloadRegion: eu-west-1
```
Broker logs:
```log
15:23:53.548 [pulsar-io-24-1] INFO
org.apache.pulsar.broker.service.ServerCnx -
[/192.168.183.79:49920][persistent://packhunt/tests/test-topic-1] Creating
producer. producerId=0
15:23:53.549 [Thread-1050] INFO org.apache.pulsar.broker.service.ServerCnx
- [/192.168.183.79:49920] persistent://packhunt/tests/test-topic-1 configured
with schema false
15:23:53.550 [Thread-1050] INFO org.apache.pulsar.broker.service.ServerCnx
- [/192.168.183.79:49920] Created new producer:
Producer{topic=PersistentTopic{topic=persistent://packhunt/tests/test-topic-1},
client=/192.168.183.79:49920, producerName=pulsar-50-1007, producerId=0}
15:23:53.603 [bookkeeper-ml-workers-OrderedExecutor-7-0] INFO
org.apache.bookkeeper.mledger.impl.OpAddEntry -
[packhunt/tests/persistent/test-topic-1] Closing ledger 2227 for being full
15:23:53.608 [bookkeeper-ml-workers-OrderedExecutor-7-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[packhunt/tests/persistent/test-topic-1] Going to automatically offload ledgers
[2215, 2216, 2217, 2218, 2219, 2220], total size = 3013527080, already
offloaded = 0, to offload = 3010342423
15:23:53.608 [bookkeeper-ml-workers-OrderedExecutor-7-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[packhunt/tests/persistent/test-topic-1] Preparing metadata to offload ledger
2215 with uuid b965b889-7202-4ada-9dd3-d9ce07b552d8
15:23:53.608 [bookkeeper-ml-workers-OrderedExecutor-7-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[packhunt/tests/persistent/test-topic-1] Found previous offload attempt for
ledger 2215, uuid b965b889-7202-4ada-9dd3-d9ce07b552d8, cleaning up
15:23:53.618 [pulsar-web-33-3] INFO org.eclipse.jetty.server.RequestLog -
192.168.182.18 - - [16/Jun/2020:15:23:53 +0000] "GET /status.html HTTP/1.1" 200
2 "-" "kube-probe/1.14+" 0
15:23:53.618 [bookkeeper-ml-workers-OrderedExecutor-7-0] WARN
org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble:
[pulsar-bookie-3.pulsar-bookie.pulsar2.svc.cluster.local:3181,
pulsar-bookie-2.pulsar-bookie.pulsar2.svc.cluster.local:3181,
pulsar-bookie-0.pulsar-bookie.pulsar2.svc.cluster.local:3181] is not adhering
to Placement Policy. quarantinedBookies: []
15:23:53.621 [pulsar-ordered-OrderedExecutor-5-0-EventThread] INFO
org.apache.pulsar.zookeeper.ZooKeeperCache - [State:CONNECTED Timeout:30000
sessionid:0x300000237e600ca local:/192.168.163.218:39918
remoteserver:pulsar-zookeeper/192.168.169.137:2181 lastZxid:4295194973
xid:14558 sent:14558 recv:15712 queuedpkts:0 pendingresp:0 queuedevents:0]
Received ZooKeeper watch event: WatchedEvent state:SyncConnected
type:NodeDataChanged
path:/managed-ledgers/packhunt/tests/persistent/test-topic-1
15:23:53.621 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO
org.apache.pulsar.zookeeper.ZooKeeperCache - [State:CONNECTED Timeout:30000
sessionid:0x200000249b200c4 local:/192.168.152.113:59370
remoteserver:pulsar-zookeeper/192.168.127.51:2181 lastZxid:4295194975 xid:15908
sent:15908 recv:17066 queuedpkts:0 pendingresp:1 queuedevents:1] Received
ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/managed-ledgers/packhunt/tests/persistent/test-topic-1
15:23:53.621 [bookkeeper-ml-workers-OrderedExecutor-7-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[packhunt/tests/persistent/test-topic-1] Metadata prepared for offload of
ledger 2215 with uuid b965b889-7202-4ada-9dd3-d9ce07b552d8
15:23:53.621 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO
org.apache.pulsar.zookeeper.ZooKeeperCache - [State:CONNECTED Timeout:30000
sessionid:0x100045f141f00d4 local:/192.168.191.251:40648
remoteserver:pulsar-zookeeper/192.168.129.222:2181 lastZxid:4295194973
xid:16249 sent:16249 recv:17421 queuedpkts:0 pendingresp:0 queuedevents:0]
Received ZooKeeper watch event: WatchedEvent state:SyncConnected
type:NodeDataChanged
path:/managed-ledgers/packhunt/tests/persistent/test-topic-1
15:23:53.625 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO
org.apache.bookkeeper.client.LedgerCreateOp - Ensemble:
[pulsar-bookie-3.pulsar-bookie.pulsar2.svc.cluster.local:3181,
pulsar-bookie-2.pulsar-bookie.pulsar2.svc.cluster.local:3181,
pulsar-bookie-0.pulsar-bookie.pulsar2.svc.cluster.local:3181] for ledger: 2233
15:23:53.625 [pulsar-ordered-OrderedExecutor-7-0-EventThread] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[packhunt/tests/persistent/test-topic-1] Created new ledger 2233
15:23:54.092 [offloader-OrderedScheduler-1-0] ERROR
org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader
- Failed delete Blob
org.jclouds.aws.AWSResponseException: request POST
https://test-offload-bucket.s3.amazonaws.com/?delete HTTP/1.1 failed with code
400, error: AWSError{requestId='031C5FE0A1D8315F',
requestToken='F3G4gdz0gL7xFVMwIR1Dqcx+tNejlS7XeeRKc65VD6TpvCUYBeuPH5wOGmQHicUPZWchEhkihQQ=',
code='AuthorizationHeaderMalformed', message='The authorization header is
malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'',
context='{Region=eu-west-1,
HostId=F3G4gdz0gL7xFVMwIR1Dqcx+tNejlS7XeeRKc65VD6TpvCUYBeuPH5wOGmQHicUPZWchEhkihQQ=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[?:?]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[?:?]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
~[?:?]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
~[?:?]
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
~[?:?]
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
~[?:?]
at com.sun.proxy.$Proxy67.deleteObjects(Unknown Source) ~[?:?]
at
org.jclouds.s3.blobstore.S3BlobStore.removeBlobs(S3BlobStore.java:343) ~[?:?]
at
org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$deleteOffloaded$6(BlobStoreManagedLedgerOffloader.java:575)
~[?:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_232]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
[com.google.guava-guava-25.1-jre.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
[com.google.guava-guava-25.1-jre.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
[com.google.guava-guava-25.1-jre.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_232]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_232]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_232]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
15:23:54.100 [bookkeeper-ml-scheduler-OrderedScheduler-7-0] WARN
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Error cleaning up
offload for 2215, (cleanup reason: Previous failed offload)
org.jclouds.aws.AWSResponseException: request POST
https://test-offload-bucket.s3.amazonaws.com/?delete HTTP/1.1 failed with code
400, error: AWSError{requestId='031C5FE0A1D8315F',
requestToken='F3G4gdz0gL7xFVMwIR1Dqcx+tNejlS7XeeRKc65VD6TpvCUYBeuPH5wOGmQHicUPZWchEhkihQQ=',
code='AuthorizationHeaderMalformed', message='The authorization header is
malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'',
context='{Region=eu-west-1,
HostId=F3G4gdz0gL7xFVMwIR1Dqcx+tNejlS7XeeRKc65VD6TpvCUYBeuPH5wOGmQHicUPZWchEhkihQQ=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[?:?]
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[?:?]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
~[?:?]
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
~[?:?]
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
~[?:?]
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
~[?:?]
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
~[?:?]
at com.sun.proxy.$Proxy67.deleteObjects(Unknown Source) ~[?:?]
at
org.jclouds.s3.blobstore.S3BlobStore.removeBlobs(S3BlobStore.java:343) ~[?:?]
at
org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$deleteOffloaded$6(BlobStoreManagedLedgerOffloader.java:575)
~[?:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_232]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
~[com.google.guava-guava-25.1-jre.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
~[com.google.guava-guava-25.1-jre.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
~[com.google.guava-guava-25.1-jre.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_232]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_232]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_232]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
