Hi Nicolas,
It seems that NiFi expects to have the "mode" field being present, even
though based on the BigQuery doc [1] it's optional.
I'd suggest trying adding it to every name-type pair with its default value
"NULLABLE". (i.e. { "name": "Consent", "type": "record", *"mode":
"NULLABLE"*, "fields": [ { "name": "id", "type": "STRING", *"mode":
"NULLABLE"* }, ...)
Let me know if it solved the issue. If yes, I'll file a Jira ticket to fix
it.
Best,
Denes
[1]
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema
On Wed, Jul 3, 2019 at 11:07 AM Nicolas Delsaux <[email protected]>
wrote:
> I'm using Apache Nifi 1.9.2 and trying to post JSON content to a
> BigQuery table.
>
> There seems to be something wrong, sicne I get
>
>
> 2019-07-03 08:35:24,964 ERROR [Timer-Driven Process Thread-8]
> o.a.n.p.gcp.bigquery.PutBigQueryBatch
> PutBigQueryBatch[id=b2b1c6bf-016b-1000-e8c9-b3f9fb5b417e] null:
> java.lang.NullPointerException
> java.lang.NullPointerException: null
> at
>
> org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.mapToField(BigQueryUtils.java:42)
> at
>
> org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.listToFields(BigQueryUtils.java:68)
> at
>
> org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.schemaFromString(BigQueryUtils.java:80)
> at
>
> org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch.onTrigger(PutBigQueryBatch.java:277)
> at
>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> at
>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Where can it come from ? And how can i fix it ?
>
>
> From the stack, I'm understanding there is something wrong with my
> BigQuery schema (which is however recognized as valid by BigQuery).
>
>
> My schema is
>
>
> [
> {
> "name": "Consent",
> "type": "record",
> "fields": [
> {
> "name": "id",
> "type": "STRING"
> },
> {
> "name": "identity",
> "type": "record",
> "fields": [
> {
> "name": "id",
> "type": "STRING"
> },
> {
> "name": "type",
> "type": "STRING"
> },
> {
> "name": "businessUnit",
> "type": "STRING"
> }
> ]
> },
> {
> "name": "finality",
> "type": "STRING"
> },
> {
> "name": "source",
> "type": "record",
> "fields": [
> {
> "name": "id",
> "type": "STRING"
> },
> {
> "name": "type",
> "type": "STRING"
> },
> {
> "name": "origin",
> "type": "STRING"
> },
> {
> "name": "collaborator",
> "type": "record",
> "fields": [
> {
> "name": "id",
> "type": "STRING"
> },
> {
> "name": "type",
> "type": "STRING"
> }
> ]
> }
> ]
> },
> {
> "name": "consentDate",
> "type": "TIMESTAMP"
> },
> {
> "name": "expiryDate",
> "type": "TIMESTAMP"
> },
> {
> "name": "expired",
> "type": "BOOLEAN"
> },
> {
> "name": "createdBy",
> "type": "STRING"
> },
> {
> "name": "createdDate",
> "type": "TIMESTAMP"
> }
> ]
> }
> ]
>
>
> What can cause the trouble ?
>
>
> Thanks
>
>
