Chih Han Yu created NIFI-8611:
---------------------------------
Summary: GCP BigQuery processors support using designate project
resource for ingestion
Key: NIFI-8611
URL: https://issues.apache.org/jira/browse/NIFI-8611
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.11.4
Reporter: Chih Han Yu
Fix For: 1.14.0
For now, `PutBigQueryBatch` processor and `PutBigQueryStreaming` processor can
only assign a single project id for consuming resources and do ingestion. But
in some business cases, the project providing resources and the project which
be inserted are not always the same.
src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
{code:java}
......
public static final PropertyDescriptor PROJECT_ID = new PropertyDescriptor
.Builder().name("gcp-project-id")
.displayName("Project ID")
.description("Google Cloud Project ID")
.required(false)
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
......{code}
We've test a solution which is workable, which is, adding another property
`DESIGNATE_PROJECT_ID` in `AbstractBigQueryProcessor`, it'll only impact
`PutBigQueryBatch` processor and `PutBigQueryStreaming` processor.
If user provides designate project id:
* Use `PROJECT_ID` (defined in `AbstractGCPProcessor*`*) as resource consuming
project.
* Put data into `DESIGNATE_PROJECT_ID` (defined in
`AbstractBigQueryProcessor*`*).
If user does not provide designate project id:
* Use `PROJECT_ID` (defined in `AbstractGCPProcessor*`*) as resource consuming
project.
* Put data into `PROJECT_ID` (defined in `AbstractGCPProcessor*`*).
Since we already have solution in production environment, I'll submit a PR
later for this improvement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
