Re: Consuming one PCollection before consuming another with Beam

large, >> you can read it from BQ and make it a side input into the DoFn. >> >> On Mon, Feb 27, 2023 at 11:06 AM Sahil Modak >> wrote: >> >>> We are trying to re-initialize our state specs in the BusinessLogic() >>> DoFn from BQ. >>> BQ has data

Re: Consuming one PCollection before consuming another with Beam

process the BigQuery data in a separate transform >> first, but it depends on the structure of the data. How large is the >> BigQuery table? Are you doing any windowed aggregations here? >> >> Reuven >> >> On Fri, Feb 24, 2023 at 10:40 AM Sahil Modak >> wro

Re: Consuming one PCollection before consuming another with Beam

s this a streaming pipeline? If > so, how are you reading from BigQuery? > > On Thu, Feb 23, 2023 at 10:06 PM Sahil Modak via dev > wrote: > >> Hi, >> >> We have a requirement wherein we are consuming input from pub/sub >> (PubSubIO) as well as BQ (BQIO) >

Consuming one PCollection before consuming another with Beam

Hi, We have a requirement wherein we are consuming input from pub/sub (PubSubIO) as well as BQ (BQIO) We want to make sure that we consume the BQ stream first before we start consuming the data from pub-sub. Is there a way to achieve this? Can you please help with some code samples? Currently,

Using unbounded source as a side input for a DoFn

Hi, We are looking to use the side input feature for one of our DoFns. The side input has to be a PCollection which is being constructed from a subscription using PubsubIO.read We want our primary DoFn which operates in a global window KV pair to access this side input. The goal is to have all

Update BEAM-11017

Hi, Is this issue (BEAM-11017 ) already fixed or is there a plan to fix this? We are also experiencing issues wherein the PROCESSING_TIME timer is not expiring in certain cases, whereas in certain cases, it's expiring early than expected. Please

Beams processing_time timer expiring early than expected

Hi, We are using beam's PROCESSING_TIME based timers in a DoFn that operates on a KV pair in a global window. We are using 2 timers in our business logic and in one of our timer's callback function, we are setting the second timer to expire after x seconds. However, the observation is that this

Re: Accessing side input in event_time timer callback function

:02 PM Sahil Modak wrote: > Hi Luke, > > Please find answers below: > 1) Number of keys in main input : this is indefinite, currently we are > observing around ~2 million keys > 2) How often (if ever) does the side input change : this will stay > constant most of the time, wil

Re: Accessing side input in event_time timer callback function

zTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo=rrh7TUJDMiiwiRP8ZZfi_SaO8JJfpCE9KWDX13mPj7k=ucnxB09gIQs1ccxLdY7p_LSO6yAQyZIOk5-KTvIFvQqFR6EP_2ultOTsi-Dxj_Cw=puwDWFS5HBrSRZWocwPFvySnWopLgW1WxQ5GJdAUfRM=> > > > On Thu, Dec 30, 2021 at 2:37 AM Sahil Modak > wrote: > >> Hi, >> >>

Accessing side input in event_time timer callback function

Hi, We are using beam's EVENT_TIME based timers in a DoFn that operates on a KV pair in a global window. We are also providing this DoFn with a side input, however we are unable to access this side input in the callback function provided for the EVENT_TIME based timers. Is there way to access

Connect to Redis with SSL

Hi, We have a beam pipeline in Java, which is supposed to read things from redis. I am trying to use RedisIO class : https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/redis/RedisIO.html However, the challenge I am facing here is redis is TLS enabled and I need to pass some