Re: Individual Parallelism support for Flink Runner

Looks like https://ci.apache.org/projects/flink/flink-docs-stable/dev/parallel.html#operator-level Regards, Amit On Mon, Jun 29, 2020 at 12:59 PM Kenneth Knowles wrote: > This exact issue has been discussed before, though I can't find the older > threads. Basically, specifying parallelism is

Re: DynamicMessage in protobufs for re-usable beam pipelines

D+Beam+Schema+Options > [2] https://github.com/apache/beam/pull/10502 > > On Mon, Jun 15, 2020 at 1:15 AM amit kumar wrote: > >> Hi, >> >> >> I intend to use Protobuf options to trigger different transforms and use >> metadata from storage proto option

DynamicMessage in protobufs for re-usable beam pipelines

Hi, I intend to use Protobuf options to trigger different transforms and use metadata from storage proto options for sink partitioning etc.. and also allow different protobuf message types flowing via the same pipeline, running as different instances of the pipeline. I am able to parse

Runner dependent sharding for dynamic destinations in FileIO

Hi Everyone, We use FileIO's writeDynamic to write dynamically to separate groups based on an attribute's value in the input PCollection. I wanted to check if there is a way to make sharding as runner dependent? Many thanks, Amit

Re: Default WindowFn for Unbounded source

Thank you all! your responses are very helpful. On Wed, Apr 1, 2020 at 11:37 AM Robert Bradshaw wrote: > > > On Wed, Apr 1, 2020 at 12:53 AM Jan Lukavský wrote: > >> Hi Amit, >> >> answers inline. >> On 4/1/20 12:23 AM, amit kumar wrote: >> >&g

Re: Default WindowFn for Unbounded source

. > > Thanks, > Ankur > > On Tue, Mar 31, 2020 at 11:20 AM amit kumar wrote: > >> Thanks Jan! >> I have a question based on this on Global Window and allowed lateness, >> with default trigger for the following >> scenarios: >> >>

Re: Default WindowFn for Unbounded source

Window |. (Will I never need to do allowedLateness in this case with default trigger? Will there be any benefit since the window is global and watermark will pass the end of window when everything is processed ? ) On Tue, Mar 31, 2020 at 11:20 AM amit kumar wrote: > Thanks Jan! >

Re: Default WindowFn for Unbounded source

[1] - global window with zero allowed > lateness. > > Cheers, > > Jan > > [1] > > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java#L105 > > On 3/31/20 10:22 AM, amit kumar wrote: > > Hi

Default WindowFn for Unbounded source

Hi All, Is there a default WindowFn that gets applied to elements of an unbounded source. For example, if I have a Kinesis input source ,for which all elements are timestamped with ArrivalTime, what will be the default windowing applied to the output of read transform ? Is this runner dependent

Discrete Transforms vs One Single transform

Hi All, I am looking for inputs to understand the effects of converting multiple discrete transforms into one single transformation. (and performing all steps into one single PTransform). What is better approach, multiple discrete transforms vs one single transform with lambdas and multiple

Re: Contributor permission for Beam Jira tickets

THanks! On Tue, Nov 12, 2019 at 3:49 PM Kenneth Knowles wrote: > Done. Welcome! > > On Tue, Nov 12, 2019 at 3:40 PM amit kumar wrote: > >> Hi Beam Devs, >> >> I am Amit from Godaddy and I am looking to contribute to Beam. >> Could you please add me as a

Re: Contributor permission for Beam Jira tickets

Hi Beam Devs, I am Amit from Godaddy and I am looking to contribute to Beam. Could you please add me as a contributor. My Id is - amitkumar27 Regards, Amit On Wed, Nov 6, 2019 at 9:59 AM amit kumar wrote: > Hi Beam Devs, > > I am Amit from Godaddy and I am looking to contribut