[
https://issues.apache.org/jira/browse/NIFI-11791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742517#comment-17742517
]
David Handermann commented on NIFI-11791:
-----------------------------------------
[~pvillard] and [~msugar] thanks for the background and evaluation of the
options. Of the possible approaches, it seems like either the PutBigQueryBatch
Processor could be restored, or a new Processor could be built that uses the
core BigQuery API without the legacy API. I'm not familiar with the
implementation details of PutBigQueryBatch, so it seems like the next step
would be to evaluate whether it makes sense to simply restore the Processor, or
create a new more-focused Processor.
> PutBigQuery processor lacks functionality found in PutBigQueryBatch
> -------------------------------------------------------------------
>
> Key: NIFI-11791
> URL: https://issues.apache.org/jira/browse/NIFI-11791
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.0.0, 1.22.0
> Reporter: Marcio Sugar
> Priority: Major
>
> Before PutBigQuery, we had PutBigQueryBatch and PutBigQueryStream, both now
> deprecated. Not sure if PutBigQuery was designed to completely replace its
> older brothers, but it cannot do that yet because of some missing features.
> For example, we can't use PubBigQuery alone to create snapshot tables,
> something that was easy to do with PutBigQueryBatch.
> A snapshot table is a recent copy of a table from a database or a subset of
> rows/columns of a table. It is used to dynamically replicate data between
> distributed databases. Using PutBigQueryBatch, we can achieve that by setting
> the following properties:
> * Create Disposition = CREATE_IF_NEEDED
> * Write Disposition = WRITE_TRUNCATE
> I understand that PutBigQuery uses the newer [BigQuery Storage Write
> API|https://cloud.google.com/bigquery/docs/write-api], so adding the missing
> functionality might not be possible.
> But please note the older BigQuery (core) API (the one I believe
> PutBigQueryBatch uses) allows the user to submit jobs to load data into
> BigQuery in a very convenient way. That is something I'd like to see
> preserved in future versions of NiFi
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
