Thanks Pierre - you rock as always
On Thu, Jan 13, 2022 at 1:12 PM Pierre Villard <[email protected]> wrote: > > Hi Franco, > > I believe this post contains useful information: > https://stackoverflow.com/questions/43085896/update-or-delete-tables-with-streaming-buffer-in-bigquery > > In short: >> >> To check if the table has a streaming buffer, check the tables.get response >> for a section named streamingBuffer or, when streaming to a partitioned >> table, data in the streaming buffer has a NULL value for the _PARTITIONTIME >> pseudo column, so even with a simple WHERE query can be checked. >> >> >> >> Streamed data is available for real-time analysis within a few seconds of >> the first streaming insertion into a table but it can take up to 90 minutes >> to become available for copy/export and other operations. You probably have >> to wait up to 90 minutes so all buffer is persisted on the cluster. You can >> use queries to see if the streaming buffer is empty or not like you >> mentioned. > > > You can also add a WHERE statement to your query to make sure it only looks > at the data that is NOT in the buffer. > > It sounds like there is a new unified API available for pushing data in > BigQuery: > https://cloud.google.com/bigquery/docs/write-api > It became GA in October 2021. I'll try to spend some time building a > processor that uses this new API (not sure the existing processor can be > updated to use the new API, but I'll check). > > Hope this helps, > Thanks, > Pierre > > > Le jeu. 13 janv. 2022 à 20:55, walter franco <[email protected]> a écrit : >> >> Thanks a lot Joe >> >> Nifi version 1.12.1 >> >> El jue, 13 ene 2022 a las 14:50, Joe Witt (<[email protected]>) escribió: >>> >>> Franco >>> >>> I've pinged someone in the community who I suspect will know or be >>> able to quickly find out and reply back. But he is in France and it >>> is already later in the day so hang tight for a reply - hopefully >>> tomorrow. >>> >>> Thanks >>> >>> On Thu, Jan 13, 2022 at 12:40 PM walter franco <[email protected]> wrote: >>> > >>> > hello, >>> > >>> > We have an error with the bigqueryStreaming processor, we send the data >>> > and it writes correctly in bigquery, but when we execute in the bigquery >>> > platform, we are getting an error response "UPDATE or DELETE statement >>> > over table (tableX) would affect rows in the streaming buffer, which is >>> > not supported" >>> > Do you think it's because of the way the bigqueryStreaming process works? >>> > We have seen that after 30 min, if the query in bigquery is successful. >>> > Is there any blockage due to the way the processor works? >>> > >>> > thanks for your help >>> > >>> > Franco
