Hi James,
Yes, 'gsutil ls' lists the bucket fine when I run it from the instance.
The service account was created and associated with the Storage Object
Viewer and the Storage Object Creator roles.
I have also added the role to the acl of the bucket (Bucket Permissions)
and objects in the bucket (Object default permissions).
gsutil iam get gs://ap_1908
{
"bindings": [
{
"members": [
"projectEditor:PROJECT_ID",
"projectOwner:PROJECT_ID",
"serviceAccount:SERVICE_ACC_ID@PROJECT_ID.iam.gserviceaccount.com"
],
"role": "roles/storage.legacyBucketOwner"
},
{
"members": [
"projectViewer:automation-platform"
],
"role": "roles/storage.legacyBucketReader"
}
],
"etag": "CAY="
}
Is there anything weird I am doing here?
Also, is there an end to end example of what all the minimum config
settings should be to run this successfully?
Thanks,
Alex
P.S. The same thing happens if just running with the default dev instance
service account.
On Fri, May 26, 2017 at 5:46 PM, James Wing <[email protected]> wrote:
> Martin,
>
> If you ran "gsutil ls" against the bucket from the instance using the
> service account, is it able to list the bucket? Can you describe a bit
> more about how the service account's access to the bucket was configured?
> Is it added to one of the project roles, explicitly granted on the bucket,
> or something else?
>
> Thanks,
>
> James
>
> On Fri, May 26, 2017 at 2:04 AM, Martin Eden <[email protected]>
> wrote:
>
> > Hi all,
> >
> > I have a Google Compute instance with a service account configured.
> >
> > I have installed Nifi 1.2.0 on it.
> >
> > In the Nifi UI I am describing a flow.
> >
> > The first processor is a ListGCSBucket processor. It is pointing to a
> > bucket for which the service account is added as owner and I am not
> setting
> > a Prefix.
> >
> > For the GCPCredentialsControllerService I have enabled, one by one all
> the
> > possible properties
> > Use Application Default Credentials
> > Use Compute Engine Credentials
> > Service Account JSON File
> > Service Account JSON
> >
> > to no avail.
> >
> > In any configuration I get the below.
> >
> > Any suggestions on how to make this work would be much appreciated,
> > M
> >
> > 2017-05-26 06:59:59,882 ERROR [Timer-Driven Process Thread-3]
> > o.a.n.p.gcp.storage.ListGCSBucket ListGCSBucket[id=xxx-xxx-xxx-xxx]
> > ListGCSBucket[id=xxx-xxx-xxx-xxx] failed to process session due to
> > com.google.cloud.storage.StorageException: Not Found: {}
> >
> > com.google.cloud.storage.StorageException: Not Found
> >
> > at
> > com.google.cloud.storage.spi.DefaultStorageRpc.translate(
> > DefaultStorageRpc.java:202)
> >
> > at
> > com.google.cloud.storage.spi.DefaultStorageRpc.list(
> > DefaultStorageRpc.java:294)
> >
> > at com.google.cloud.storage.StorageImpl$8.call(
> > StorageImpl.java:297)
> >
> > at com.google.cloud.storage.StorageImpl$8.call(
> > StorageImpl.java:294)
> >
> > at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:179)
> >
> > at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.
> > java:244)
> >
> > at
> > com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:293)
> >
> > at com.google.cloud.storage.StorageImpl.list(StorageImpl.
> java:260)
> >
> > at
> > org.apache.nifi.processors.gcp.storage.ListGCSBucket.
> > onTrigger(ListGCSBucket.java:262)
> >
> > at
> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> > AbstractProcessor.java:27)
> >
> > at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> > StandardProcessorNode.java:1118)
> >
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> > ContinuallyRunProcessorTask.java:144)
> >
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> > ContinuallyRunProcessorTask.java:47)
> >
> > at
> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> > TimerDrivenSchedulingAgent.java:132)
> >
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >
> > at java.util.concurrent.FutureTask.runAndReset(
> > FutureTask.java:308)
> >
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(
> > ScheduledThreadPoolExecutor.java:294)
> >
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> >
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> >
> > at java.lang.Thread.run(Thread.java:748)
> >
> > Caused by:
> > com.google.api.client.googleapis.json.GoogleJsonResponseException: 404
> Not
> > Found
> >
> > {
> >
> > "code" : 404,
> >
> > "errors" : [ {
> >
> > "domain" : "global",
> >
> > "message" : "Not Found",
> >
> > "reason" : "notFound"
> >
> > } ],
> >
> > "message" : "Not Found"
> >
> > }
> >
> > at
> > com.google.api.client.googleapis.json.GoogleJsonResponseException.from(
> > GoogleJsonResponseException.java:146)
> >
> > at
> > com.google.api.client.googleapis.services.json.
> > AbstractGoogleJsonClientRequest.newExceptionOnError(
> > AbstractGoogleJsonClientRequest.java:113)
> >
> > at
> > com.google.api.client.googleapis.services.json.
> > AbstractGoogleJsonClientRequest.newExceptionOnError(
> > AbstractGoogleJsonClientRequest.java:40)
> >
> > at
> > com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.
> > interceptResponse(AbstractGoogleClientRequest.java:321)
> >
> > at
> > com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1049)
> >
> > at
> > com.google.api.client.googleapis.services.AbstractGoogleClientRequest.
> > executeUnparsed(AbstractGoogleClientRequest.java:419)
> >
> > at
> > com.google.api.client.googleapis.services.AbstractGoogleClientRequest.
> > executeUnparsed(AbstractGoogleClientRequest.java:352)
> >
> > at
> > com.google.api.client.googleapis.services.AbstractGoogleClientRequest.
> > execute(AbstractGoogleClientRequest.java:469)
> >
> > at
> > com.google.cloud.storage.spi.DefaultStorageRpc.list(
> > DefaultStorageRpc.java:277)
> >
> > ... 19 common frames omitted
> >
>
