Re: How does groupIntoBatches behave when there are too few elements for a key?

2020-08-26 Thread Siyuan Chen
I have been preparing a PR to add the timeout option. I had a dumb question - seems to me that the timeout should be set in processing time while the existing timer fired at the window expiration is in event time. Is there a way to have timers in different time domains? -- Best regards, Siyuan

Re: Composable DoFn IOs Connection Reuse

2020-07-13 Thread Siyuan Chen
gt;> elements and doing something with all of them together at once (e.g. >>> service call). >>> >>> >>> On Mon, Jun 29, 2020 at 3:00 PM Siyuan Chen wrote: >>> >>>> >>>> -- >>>> Best regards, >>>> Siyuan

Contributor permission for Beam Jira tickets

2020-07-13 Thread Siyuan Chen
Hi, This is Siyuan from Google. I am working on scalability improvements for Dataflow runner. Can someone grant me the contributor permission for Beam Jira tickets? My Jira username is sychen. Thanks in advance! -- Best regards, Siyuan

Re: Composable DoFn IOs Connection Reuse

2020-06-29 Thread Siyuan Chen
ve >>> been useful for the Cloud AI transforms recently introduced. >>> >>> On Fri, Jun 26, 2020 at 3:25 PM Tyson Hamilton >>> wrote: >>> >>>> >>>> >>>> On Fri, Jun 26, 2020 at 12:24 PM Siyuan Chen wrote: >&g

Re: Composable DoFn IOs Connection Reuse

2020-06-26 Thread Siyuan Chen
he same "key" > again without limiting the number of workers that can process a specific > key. This is very similar to what we do with a stateful DoFn but one where > the runner knows that it can "shard" the key. +Siyuan Chen > has been investigating something like