Hi Franco, I believe this post contains useful information: https://stackoverflow.com/questions/43085896/update-or-delete-tables-with-streaming-buffer-in-bigquery
In short: > To check if the table has a streaming buffer, check the tables.get response > for a section named streamingBuffer or, when streaming to a partitioned > table, data in the streaming buffer has a NULL value for the > _PARTITIONTIME pseudo column, so even with a simple WHERE query can be > checked. Streamed data > <https://cloud.google.com/bigquery/streaming-data-into-bigquery> is > available for real-time analysis within a few seconds of the first > streaming insertion into a table but it can take up to 90 minutes to become > available for copy/export and other operations. You probably have to wait > up to 90 minutes so all buffer is persisted on the cluster. You can use > queries to see if the streaming buffer is empty or not like you mentioned. You can also add a WHERE statement to your query to make sure it only looks at the data that is NOT in the buffer. It sounds like there is a new unified API available for pushing data in BigQuery: https://cloud.google.com/bigquery/docs/write-api It became GA in October 2021. I'll try to spend some time building a processor that uses this new API (not sure the existing processor can be updated to use the new API, but I'll check). Hope this helps, Thanks, Pierre Le jeu. 13 janv. 2022 à 20:55, walter franco <[email protected]> a écrit : > Thanks a lot Joe > > Nifi version 1.12.1 > > El jue, 13 ene 2022 a las 14:50, Joe Witt (<[email protected]>) escribió: > >> Franco >> >> I've pinged someone in the community who I suspect will know or be >> able to quickly find out and reply back. But he is in France and it >> is already later in the day so hang tight for a reply - hopefully >> tomorrow. >> >> Thanks >> >> On Thu, Jan 13, 2022 at 12:40 PM walter franco <[email protected]> >> wrote: >> > >> > hello, >> > >> > We have an error with the bigqueryStreaming processor, we send the data >> and it writes correctly in bigquery, but when we execute in the bigquery >> platform, we are getting an error response "UPDATE or DELETE statement over >> table (tableX) would affect rows in the streaming buffer, which is not >> supported" >> > Do you think it's because of the way the bigqueryStreaming process >> works? >> > We have seen that after 30 min, if the query in bigquery is successful. >> Is there any blockage due to the way the processor works? >> > >> > thanks for your help >> > >> > Franco >> >
