I'm ivnestigating the same way.
I've added the mode field everywhere, but still have the issue.
I'll try to create a minimal reproducing schema for your ticket (by
running unit tests)
Le 03/07/2019 à 11:28, Denes Arvay a écrit :
Hi Nicolas,
It seems that NiFi expects to have the "mode" field being present,
even though based on the BigQuery doc [1] it's optional.
I'd suggest trying adding it to every name-type pair with its default
value "NULLABLE". (i.e. { "name": "Consent", "type": "record",
*"mode": "NULLABLE"*, "fields": [ { "name": "id", "type": "STRING",
*"mode": "NULLABLE"* }, ...)
Let me know if it solved the issue. If yes, I'll file a Jira ticket to
fix it.
Best,
Denes
[1]
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema
On Wed, Jul 3, 2019 at 11:07 AM Nicolas Delsaux
<nicolas.dels...@gmx.fr <mailto:nicolas.dels...@gmx.fr>> wrote:
I'm using Apache Nifi 1.9.2 and trying to post JSON content to a
BigQuery table.
There seems to be something wrong, sicne I get
2019-07-03 08:35:24,964 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.gcp.bigquery.PutBigQueryBatch
PutBigQueryBatch[id=b2b1c6bf-016b-1000-e8c9-b3f9fb5b417e] null:
java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.mapToField(BigQueryUtils.java:42)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.listToFields(BigQueryUtils.java:68)
at
org.apache.nifi.processors.gcp.bigquery.BigQueryUtils.schemaFromString(BigQueryUtils.java:80)
at
org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch.onTrigger(PutBigQueryBatch.java:277)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Where can it come from ? And how can i fix it ?
From the stack, I'm understanding there is something wrong with my
BigQuery schema (which is however recognized as valid by BigQuery).
My schema is
[
{
"name": "Consent",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "identity",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "businessUnit",
"type": "STRING"
}
]
},
{
"name": "finality",
"type": "STRING"
},
{
"name": "source",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
},
{
"name": "origin",
"type": "STRING"
},
{
"name": "collaborator",
"type": "record",
"fields": [
{
"name": "id",
"type": "STRING"
},
{
"name": "type",
"type": "STRING"
}
]
}
]
},
{
"name": "consentDate",
"type": "TIMESTAMP"
},
{
"name": "expiryDate",
"type": "TIMESTAMP"
},
{
"name": "expired",
"type": "BOOLEAN"
},
{
"name": "createdBy",
"type": "STRING"
},
{
"name": "createdDate",
"type": "TIMESTAMP"
}
]
}
]
What can cause the trouble ?
Thanks
