What if you run the below command in Cloud Shell:
bq --format=prettyjson show -j <job id>
In your case (with your last email):
bq --format=prettyjson show -j 9e790299-dc77-46f4-8978-476f284fe5b5
Does it give you more details?
Le jeu. 26 sept. 2019 à 12:13, Nicolas Delsaux <nicolas.dels...@gmx.fr> a
écrit :
> Sorry for the late reply.
>
> As of today, the issue is still present.
>
> Nifi Web UI just shows the message "Error while reading data, error
> message: JSON table encountered too many errors, giving up. Rows: 1;
> errors: 1. Please look into the errors[] collection for more details."
>
> But the log is clearer :
>
>
> --------------------------------------------------
> Standard FlowFile Attributes
> Key: 'entryDate'
> Value: 'Thu Sep 26 09:53:49 UTC 2019'
> Key: 'lineageStartDate'
> Value: 'Thu Sep 26 09:53:49 UTC 2019'
> Key: 'fileSize'
> Value: '999'
> FlowFile Attribute Map Content
> Key: 'avro.schema'
> Value:
> '{"type":"record","name":"nifiRecord","namespace":"org.apache.nifi","fields":[{"name":"ExplicitConsent","type":["null",{"type":"record","name":"ExplicitConsentType","fields":[{"name":"id","type":["null","string"]},{"name":"identity","type":["null",{"type":"record","name":"identityType","fields":[{"name":"id","type":["null","string"]},{"name":"type","type":["null","string"]},{"name":"businessUnit","type":["null","string"]}]}]},{"name":"finality","type":["null","string"]},{"name":"expired","type":["null","boolean"]},{"name":"createdBy","type":["null","string"]},{"name":"createdDate","type":["null","string"]},{"name":"sender","type":["null",{"type":"record","name":"senderType","fields":[{"name":"id","type":["null","string"]},{"name":"application","type":["null","string"]},{"name":"type","type":["null","string"]}]}]},{"name":"state","type":["null","string"]}]}]}]}'
> Key: 'bq.error.message'
> Value: 'Error while reading data, error message: JSON table encountered
> too many errors, giving up. Rows: 1; errors: 1. Please look into the
> errors[] collection for more details.'
> Key: 'bq.error.reason'
> Value: 'invalid'
> Key: 'bq.job.link'
> Value:
> 'https://www.googleapis.com/bigquery/v2/projects/psh-datacompliance/jobs/9e790299-dc77-46f4-8978-476f284fe5b5?location=EU'
>
> Key: 'bq.job.stat.creation_time'
> Value: '1569491661818'
> Key: 'bq.job.stat.end_time'
> Value: '1569491662935'
> Key: 'bq.job.stat.start_time'
> Value: '1569491662366'
> Key: 'filename'
> Value: 'e6d604d7-b517-4a87-a398-e4a5df342ce6'
> Key: 'kafka.key'
> Value: '--'
> Key: 'kafka.partition'
> Value: '0'
> Key: 'kafka.topic'
> Value: 'dc.consent-life-cycle.kpi-from-dev-nifi-json'
> Key: 'merge.bin.age'
> Value: '1'
> Key: 'merge.count'
> Value: '3'
> Key: 'mime.type'
> Value: 'application/json'
> Key: 'path'
> Value: './'
> Key: 'record.count'
> Value: '3'
> Key: 'uuid'
> Value: 'e6d604d7-b517-4a87-a398-e4a5df342ce6'
> 2019-09-26 10:09:39,633 INFO [Timer-Driven Process Thread-4]
> o.a.n.processors.standard.LogAttribute
> LogAttribute[id=ce9c171f-0c8f-3cab-e0f2-16156faf15b8] logging for flow file
> StandardFlowFileRecord[uuid=e6d604d7-b517-4a87-a398-e4a5df342ce6,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1569490848560-6, container=default,
> section=6], offset=569098,
> length=999],offset=0,name=e6d604d7-b517-4a87-a398-e4a5df342ce6,size=999]
>
>
> I don't exactly understand why i would have to set an authentication,
> because I've set the service.json content into the GCP Credentials Provider
> I use for my PutBigQueryBatch processor ...
>
> Is there anything I'm missing ? or a simple way to make sure verything
> work as expected ?
>
> Thanks
>
>
> Le 24/09/2019 à 16:12, Pierre Villard a écrit :
>
> Hey Nicolas,
>
> Did you manage to solve your issue? Happy to help on this one.
>
> Thanks,
> Pierre
>
> Le ven. 20 sept. 2019 à 16:42, Nicolas Delsaux <nicolas.dels...@gmx.fr> a
> écrit :
>
>> Hello
>>
>> I'm using PutBigQueryBash and having weird auth issues.
>>
>> I have set the GCP Credentials Controller Service to use Service Account
>> JSON which I have copied from the value given in Google Cloud Console.
>>
>> But when I run my flow, I get the error message "Error while reading
>> data, error message: JSON table encountered too many errors, giving up.
>> Rows: 1; errors: 1. Please look into the errors[] collection for more
>> details."
>>
>>
>> What is stranger is that when I log all properties, there is a
>> bq.job.link which messages indicate "Request is missing required
>> authentication credential. Expected OAuth 2 access token, login cookie or
>> other valid authentication credential. See
>> https://developers.google.com/identity/sign-in/web/devconsole-project.";
>> ...
>>
>> But nifi can access the bigquery workspace and dataset (I've checked that
>> by deleting the table schema that I have already written).
>>
>> So, is there something I'm doing wrong ?
>>
>> Thanks !
>>
>
