GitHub user devinbost created a discussion: GCP Tiered Storage
org.jclouds.rest.ResourceNotFoundException - Pulsar 2.6.1
I have a heartbeat topic that receives messages on the topic
`persistent://public/default/canary`, and I configured retention on this topic
to test tiered storage in GCP. When I manually trigger offloading from the
topic, I get the error `java.util.concurrent.CompletionException:
org.jclouds.rest.ResourceNotFoundException: Not Found`, as shown below:
`bin/pulsar-admin topics offload --size-threshold 10M
persistent://public/default/canary`
> Offload triggered for persistent://public/default/canary for messages before
> 873151:0:-1
`bin/pulsar-admin topics offload-status -w persistent://public/default/canary`
> Error in offload
> null
>
> Reason: Error offloading:
> org.apache.bookkeeper.mledger.ManagedLedgerException:
> java.util.concurrent.CompletionException:
> org.jclouds.rest.ResourceNotFoundException: Not Found
On the broker and bookie containers, I have the GCP JSON key file located at
`conf/auth/tiered-storage.key`
In my broker.conf file, I have the GCS attributes configured as follows:
```
managedLedgerOffloadDriver=google-cloud-storage
gcsManagedLedgerOffloadRegion=us-west-3
gcsManagedLedgerOffloadBucket=gs://my-bucket-name
gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864
gcsManagedLedgerOffloadReadBufferSizeInBytes=1048576
gcsManagedLedgerOffloadServiceAccountKeyFile=/pulsar/conf/auth/tiered-storage.key
```
The tiered-storage.key file looks as follows:
```
{
"type": "service_account",
"project_id": "my-project-id",
"private_key_id": "c8myid12345",
"private_key": "-----BEGIN PRIVATE KEY-----\nABCDEFG...XYZ=\n-----END PRIVATE
KEY-----\n",
"client_email": "[email protected]",
"client_id": "123456",
"auth_uri": "https://accounts.google.com/o/oauth2/auth";,
"token_uri": "https://oauth2.googleapis.com/token";,
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs";,
"client_x509_cert_url":
"https://www.googleapis.com/robot/v1/metadata/x509/example-cert-url%40identity-example.iam.gserviceaccount.com";
}
```
It seems that the key file path is incorrect. However, when I tried altering
the key file to contain invalid JSON, I got an error message that the JSON was
invalid when I triggered a manual offload of the topic, which implies that
Pulsar was able to access the key file.
GitHub link: https://github.com/apache/pulsar/discussions/18868
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
