[
https://issues.apache.org/jira/browse/NIFI-11791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742500#comment-17742500
]
Pierre Villard commented on NIFI-11791:
---------------------------------------
I guess the point I was making is that I don't see a clean approach to use the
core API in the new processor and provide the same features as the ones you
have in the current Batch/Job oriented processors. The reason is that the new
API is unified for both Batch and Stream use cases but relies on the concept of
stream to which the added is appended. It goes against the concept of job.
Hence my suggestion to not remove PutBigQueryBatch processor in NiFi 2.0.
We would likely want to build additional processors to interact with BigQuery
anyway, like executing arbitrary SQL queries without the need to go through
JDBC.
cc [~exceptionfactory] for awareness
> PutBigQuery processor lacks functionality found in PutBigQueryBatch
> -------------------------------------------------------------------
>
> Key: NIFI-11791
> URL: https://issues.apache.org/jira/browse/NIFI-11791
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.0.0, 1.22.0
> Reporter: Marcio S.
> Priority: Major
>
> Before PutBigQuery, we had PutBigQueryBatch and PutBigQueryStream, both now
> deprecated. Not sure if PutBigQuery was designed to completely replace its
> older brothers, but it cannot do that yet because of some missing features.
> For example, we can't use PubBigQuery alone to create snapshot tables,
> something that was easy to do with PutBigQueryBatch.
> A snapshot table is a recent copy of a table from a database or a subset of
> rows/columns of a table. It is used to dynamically replicate data between
> distributed databases. Using PutBigQueryBatch, we can achieve that by setting
> the following properties:
> * Create Disposition = CREATE_IF_NEEDED
> * Write Disposition = WRITE_TRUNCATE
> I understand that PutBigQuery uses the newer [BigQuery Storage Write
> API|https://cloud.google.com/bigquery/docs/write-api], so adding the missing
> functionality might not be possible.
> But please note the older BigQuery (core) API (the one I believe
> PutBigQueryBatch uses) allows the user to submit jobs to load data into
> BigQuery in a very convenient way. That is something I'd like to see
> preserved in future versions of NiFi
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
