[
https://issues.apache.org/jira/browse/NIFI-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chih Han Yu updated NIFI-8611:
------------------------------
Description:
For now, *PutBigQueryBatch* processor and *PutBigQueryStreaming* processor can
only assign a single project id for consuming resources and do ingestion. But
in some business cases, the project providing resources and the project which
be inserted are not always the same.
src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
{code:java}
......
public static final PropertyDescriptor PROJECT_ID = new PropertyDescriptor
.Builder().name("gcp-project-id")
.displayName("Project ID")
.description("Google Cloud Project ID")
.required(false)
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
......{code}
We've test a solution which is workable, which is, adding another property
*DESIGNATE_PROJECT_ID* in *AbstractBigQueryProcessor*, it'll only impact
*PutBigQueryBatch* processor and *PutBigQueryStreaming* processor.
If user provides designate project id:
* Use *PROJECT_ID* (defined in AbstractGCPProcessor) as resource consuming
project.
* Put data into *DESIGNATE_PROJECT_ID* (defined in
AbstractBigQueryProcessor).
If user does {color:#FF0000}not{color} provide designate project id:
* Use *PROJECT_ID* (defined in AbstractGCPProcessor) as resource consuming
project.
* Put data into *PROJECT_ID* (defined in AbstractGCPProcessor).
Since we already have solution in production environment, I'll submit a PR
later for this improvement.
was:
For now, `PutBigQueryBatch` processor and `PutBigQueryStreaming` processor can
only assign a single project id for consuming resources and do ingestion. But
in some business cases, the project providing resources and the project which
be inserted are not always the same.
src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
{code:java}
......
public static final PropertyDescriptor PROJECT_ID = new PropertyDescriptor
.Builder().name("gcp-project-id")
.displayName("Project ID")
.description("Google Cloud Project ID")
.required(false)
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
......{code}
We've test a solution which is workable, which is, adding another property
`DESIGNATE_PROJECT_ID` in `AbstractBigQueryProcessor`, it'll only impact
`PutBigQueryBatch` processor and `PutBigQueryStreaming` processor.
If user provides designate project id:
* Use `PROJECT_ID` (defined in `AbstractGCPProcessor*`*) as resource consuming
project.
* Put data into `DESIGNATE_PROJECT_ID` (defined in
`AbstractBigQueryProcessor*`*).
If user does not provide designate project id:
* Use `PROJECT_ID` (defined in `AbstractGCPProcessor*`*) as resource consuming
project.
* Put data into `PROJECT_ID` (defined in `AbstractGCPProcessor*`*).
Since we already have solution in production environment, I'll submit a PR
later for this improvement.
> GCP BigQuery processors support using designate project resource for ingestion
> ------------------------------------------------------------------------------
>
> Key: NIFI-8611
> URL: https://issues.apache.org/jira/browse/NIFI-8611
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.11.4
> Reporter: Chih Han Yu
> Priority: Major
> Labels: GCP, bigquery
> Fix For: 1.14.0
>
>
> For now, *PutBigQueryBatch* processor and *PutBigQueryStreaming* processor
> can only assign a single project id for consuming resources and do ingestion.
> But in some business cases, the project providing resources and the project
> which be inserted are not always the same.
> src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
>
> {code:java}
> ......
> public static final PropertyDescriptor PROJECT_ID = new PropertyDescriptor
> .Builder().name("gcp-project-id")
> .displayName("Project ID")
> .description("Google Cloud Project ID")
> .required(false)
>
> .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
> .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .build();
> ......{code}
>
> We've test a solution which is workable, which is, adding another property
> *DESIGNATE_PROJECT_ID* in *AbstractBigQueryProcessor*, it'll only impact
> *PutBigQueryBatch* processor and *PutBigQueryStreaming* processor.
> If user provides designate project id:
> * Use *PROJECT_ID* (defined in AbstractGCPProcessor) as resource consuming
> project.
> * Put data into *DESIGNATE_PROJECT_ID* (defined in
> AbstractBigQueryProcessor).
> If user does {color:#FF0000}not{color} provide designate project id:
> * Use *PROJECT_ID* (defined in AbstractGCPProcessor) as resource consuming
> project.
> * Put data into *PROJECT_ID* (defined in AbstractGCPProcessor).
> Since we already have solution in production environment, I'll submit a PR
> later for this improvement.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
