Re: BigQueryIO streaming inserts - poor performance with multiple tables

2018-03-06 Thread Carlos Alonso
Could you please keep writing here the findings you make? I'm very interested in this issue as well. Thanks! On Thu, Mar 1, 2018 at 9:45 AM Josh wrote: > Hi Cham, > > Thanks, I have emailed the dataflow-feedback email address with the > details. > > Best regards, > Josh > > On Thu, Mar 1, 2018

Re: WriteTOBigQuery/BatchLoads/ReifyResults step taking hours

2018-03-06 Thread Andrew Jones
Thanks Eugene. As you suggested, using withHintMatchesManyFiles() did result in a very significant performance increase! Enough that it's fast enough for our current use case. Will track the JIRA for any further fixes. Thanks, Andrew On Mon, 5 Mar 2018, at 22:34, Eugene Kirpichov wrote: > Filed

Re: The problem of kafkaIO sdk for data latency

2018-03-06 Thread Raghu Angadi
This message was sent with Gmail's confidential mode. You can open it by clicking this link for user@beam.apache.org.

Re: WriteTOBigQuery/BatchLoads/ReifyResults step taking hours

2018-03-06 Thread Eugene Kirpichov
Thanks, I'm glad it worked so well! I'm curious, just how much faster did it get? Do you have a job ID with the new code I can take a peek at? On Tue, Mar 6, 2018 at 4:45 AM Andrew Jones wrote: > Thanks Eugene. > > As you suggested, using withHintMatchesManyFiles() did result in a very > signifi