Having data ingestion work without needing to worry about how big the blobs are would be nice if it was automatic for users.
On Mon, Nov 12, 2018 at 1:03 AM Wout Scheepers < wout.scheep...@vente-exclusive.com> wrote: > Hey all, > > > > The TableRow size limit is 1mb when streaming into bigquery. > > To prevent data loss, I’m going to implement a TableRow size check and add > a fan out to do a bigquery load job in case the size is above the limit. > > Of course this load job would be windowed. > > > > I know it doesn’t make sense to stream data bigger than 1mb, but as we’re > using pub sub and want to make sure no data loss happens whatsoever, I’ll > need to implement it. > > > > Is this functionality any of you would like to see in BigqueryIO itself? > > Or do you think my use case is too specific and implementing my solution > around BigqueryIO will suffice. > > > > Thanks for your thoughts, > > Wout > > > > >
