Sorry for the late reply.
As of today, the issue is still present.
Nifi Web UI just shows the message "Error while reading data, error
message: JSON table encountered too many errors, giving up. Rows: 1;
errors: 1. Please look into the errors[] collection for more details."
But the log is clearer :
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Thu Sep 26 09:53:49 UTC 2019'
Key: 'lineageStartDate'
Value: 'Thu Sep 26 09:53:49 UTC 2019'
Key: 'fileSize'
Value: '999'
FlowFile Attribute Map Content
Key: 'avro.schema'
Value:
'{"type":"record","name":"nifiRecord","namespace":"org.apache.nifi","fields":[{"name":"ExplicitConsent","type":["null",{"type":"record","name":"ExplicitConsentType","fields":[{"name":"id","type":["null","string"]},{"name":"identity","type":["null",{"type":"record","name":"identityType","fields":[{"name":"id","type":["null","string"]},{"name":"type","type":["null","string"]},{"name":"businessUnit","type":["null","string"]}]}]},{"name":"finality","type":["null","string"]},{"name":"expired","type":["null","boolean"]},{"name":"createdBy","type":["null","string"]},{"name":"createdDate","type":["null","string"]},{"name":"sender","type":["null",{"type":"record","name":"senderType","fields":[{"name":"id","type":["null","string"]},{"name":"application","type":["null","string"]},{"name":"type","type":["null","string"]}]}]},{"name":"state","type":["null","string"]}]}]}]}'
Key: 'bq.error.message'
Value: 'Error while reading data, error message: JSON table encountered
too many errors, giving up. Rows: 1; errors: 1. Please look into the
errors[] collection for more details.'
Key: 'bq.error.reason'
Value: 'invalid'
Key: 'bq.job.link'
Value:
'https://www.googleapis.com/bigquery/v2/projects/psh-datacompliance/jobs/9e790299-dc77-46f4-8978-476f284fe5b5?location=EU'
Key: 'bq.job.stat.creation_time'
Value: '1569491661818'
Key: 'bq.job.stat.end_time'
Value: '1569491662935'
Key: 'bq.job.stat.start_time'
Value: '1569491662366'
Key: 'filename'
Value: 'e6d604d7-b517-4a87-a398-e4a5df342ce6'
Key: 'kafka.key'
Value: '--'
Key: 'kafka.partition'
Value: '0'
Key: 'kafka.topic'
Value: 'dc.consent-life-cycle.kpi-from-dev-nifi-json'
Key: 'merge.bin.age'
Value: '1'
Key: 'merge.count'
Value: '3'
Key: 'mime.type'
Value: 'application/json'
Key: 'path'
Value: './'
Key: 'record.count'
Value: '3'
Key: 'uuid'
Value: 'e6d604d7-b517-4a87-a398-e4a5df342ce6'
2019-09-26 10:09:39,633 INFO [Timer-Driven Process Thread-4]
o.a.n.processors.standard.LogAttribute
LogAttribute[id=ce9c171f-0c8f-3cab-e0f2-16156faf15b8] logging for flow
file
StandardFlowFileRecord[uuid=e6d604d7-b517-4a87-a398-e4a5df342ce6,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1569490848560-6,
container=default, section=6], offset=569098,
length=999],offset=0,name=e6d604d7-b517-4a87-a398-e4a5df342ce6,size=999]
I don't exactly understand why i would have to set an authentication,
because I've set the service.json content into the GCP Credentials
Provider I use for my PutBigQueryBatch processor ...
Is there anything I'm missing ? or a simple way to make sure verything
work as expected ?
Thanks
Le 24/09/2019 à 16:12, Pierre Villard a écrit :
Hey Nicolas,
Did you manage to solve your issue? Happy to help on this one.
Thanks,
Pierre
Le ven. 20 sept. 2019 à 16:42, Nicolas Delsaux <[email protected]
<mailto:[email protected]>> a écrit :
Hello
I'm using PutBigQueryBash and having weird auth issues.
I have set the GCP Credentials Controller Service to use Service
Account JSON which I have copied from the value given in Google
Cloud Console.
But when I run my flow, I get the error message "Error while
reading data, error message: JSON table encountered too many
errors, giving up. Rows: 1; errors: 1. Please look into the
errors[] collection for more details."
What is stranger is that when I log all properties, there is a
bq.job.link which messages indicate "Request is missing required
authentication credential. Expected OAuth 2 access token, login
cookie or other valid authentication credential. See
https://developers.google.com/identity/sign-in/web/devconsole-project.";
...
But nifi can access the bigquery workspace and dataset (I've
checked that by deleting the table schema that I have already
written).
So, is there something I'm doing wrong ?
Thanks !
