Thanks Pierre for your help El jue., 13 de enero de 2022 3:12 p. m., Pierre Villard < [email protected]> escribió:
> Hi Franco, > > I believe this post contains useful information: > > https://stackoverflow.com/questions/43085896/update-or-delete-tables-with-streaming-buffer-in-bigquery > > In short: > >> To check if the table has a streaming buffer, check the tables.get response >> for a section named streamingBuffer or, when streaming to a partitioned >> table, data in the streaming buffer has a NULL value for the >> _PARTITIONTIME pseudo column, so even with a simple WHERE query can be >> checked. > > > > Streamed data >> <https://cloud.google.com/bigquery/streaming-data-into-bigquery> is >> available for real-time analysis within a few seconds of the first >> streaming insertion into a table but it can take up to 90 minutes to become >> available for copy/export and other operations. You probably have to wait >> up to 90 minutes so all buffer is persisted on the cluster. You can use >> queries to see if the streaming buffer is empty or not like you mentioned. > > > You can also add a WHERE statement to your query to make sure it only > looks at the data that is NOT in the buffer. > > It sounds like there is a new unified API available for pushing data in > BigQuery: > https://cloud.google.com/bigquery/docs/write-api > It became GA in October 2021. I'll try to spend some time building a > processor that uses this new API (not sure the existing processor can be > updated to use the new API, but I'll check). > > Hope this helps, > Thanks, > Pierre > > > Le jeu. 13 janv. 2022 à 20:55, walter franco <[email protected]> a > écrit : > >> Thanks a lot Joe >> >> Nifi version 1.12.1 >> >> El jue, 13 ene 2022 a las 14:50, Joe Witt (<[email protected]>) >> escribió: >> >>> Franco >>> >>> I've pinged someone in the community who I suspect will know or be >>> able to quickly find out and reply back. But he is in France and it >>> is already later in the day so hang tight for a reply - hopefully >>> tomorrow. >>> >>> Thanks >>> >>> On Thu, Jan 13, 2022 at 12:40 PM walter franco <[email protected]> >>> wrote: >>> > >>> > hello, >>> > >>> > We have an error with the bigqueryStreaming processor, we send the >>> data and it writes correctly in bigquery, but when we execute in the >>> bigquery platform, we are getting an error response "UPDATE or DELETE >>> statement over table (tableX) would affect rows in the streaming buffer, >>> which is not supported" >>> > Do you think it's because of the way the bigqueryStreaming process >>> works? >>> > We have seen that after 30 min, if the query in bigquery is >>> successful. Is there any blockage due to the way the processor works? >>> > >>> > thanks for your help >>> > >>> > Franco >>> >>
