Re: Any recomendation for key for GroupIntoBatches
How about just keeping track of a buffer and flush the buffer after 100 messages and if there is a buffer on finish_bundle as well? On Fri, Apr 12, 2024 at 21.23 Ruben Vargas wrote: > Hello guys > > Maybe this question was already answered, but I cannot find it and > want some more input on this topic. > > I have some messages that don't have any particular key candidate, > except the ID, but I don't want to use it because the idea is to > group multiple IDs in the same batch. > > This is my use case: > > I have an endpoint where I'm gonna send the message ID, this endpoint > is gonna return me certain information which I will use to enrich my > message. In order to avoid fetching the endpoint per message I want to > batch it in 100 and send the 100 IDs in one request ( the endpoint > supports it) . I was thinking on using GroupIntoBatches. > > - If I choose the ID as the key, my understanding is that it won't > work in the way I want (because it will form batches of the same ID). > - Use a constant will be a problem for parallelism, is that correct? > > Then my question is, what should I use as a key? Maybe something > regarding the timestamp? so I can have groups of messages that arrive > at a certain second? > > Any suggestions would be appreciated > > Thanks. >
Re: Cross Language Runtime error python-Java
Adding the following to both Dockerfile.launcher and Dockerfile.worker did
the trick for me.
COPY --from=apache/beam_java11_sdk:latest /opt/apache/beam/jars
/opt/apache/beam/jars COPY --from=apache/beam_java11_sdk:latest
/opt/java/openjdk /opt/java/openjdk ENV JAVA_HOME=/opt/java/openjdk ENV
PATH="${JAVA_HOME}/bin:${PATH}"
On Sat, Feb 24, 2024 at 21:55 XQ Hu via user wrote:
> Does your code work without the launcher? Better check this step by step
> to figure out which part causes this error.
>
> On Sat, Feb 24, 2024 at 3:25 AM George Dekermenjian
> wrote:
>
>> I have a python pipeline that uses the bigquery storage write method
>> (cross language with Java). I’m building launcher and worker docker images
>> and then launching the flex template. The launcher fails due to the
>> following runtime error in dataflow.
>>
>> I’m using runner v2 and it is a streaming pipeline using the streaming
>> engine.
>>
>> Any ideas of what is causing this?
>>
>> RuntimeError:
>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>> org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>> java.lang.ClassCastException: class
>> org.apache.beam.sdk.transforms.windowing.Repeatedly cannot be cast to class
>> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger
>> (org.apache.beam.sdk.transforms.windowing.Repeatedly and
>> org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger are in unnamed
>> module of loader 'app')"
>> line: "exec.go:66"
>>
>>
Cross Language Runtime error python-Java
I have a python pipeline that uses the bigquery storage write method (cross language with Java). I’m building launcher and worker docker images and then launching the flex template. The launcher fails due to the following runtime error in dataflow. I’m using runner v2 and it is a streaming pipeline using the streaming engine. Any ideas of what is causing this? RuntimeError: org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.ClassCastException: class org.apache.beam.sdk.transforms.windowing.Repeatedly cannot be cast to class org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger (org.apache.beam.sdk.transforms.windowing.Repeatedly and org.apache.beam.sdk.transforms.windowing.Trigger$OnceTrigger are in unnamed module of loader 'app')" line: "exec.go:66"
Re: Request to join slack channel
Me too please - slack channel. On Wed, Feb 21, 2024 at 19:43 Geddy Schellevis wrote: > Hey Valentyn, > > can you add me to slack channel as well? > Thanks > > > Op wo 21 feb 2024 om 19:21 schreef Valentyn Tymofieiev via user < > user@beam.apache.org> > >> Hi Daniel, I submitted an invitation for your email address as well. >> >> On Tue, Feb 20, 2024 at 3:56 PM Daniel Chen wrote: >> >>> Hey Valentyn, can you add me to slack channel as well? >>> >>> On Tue, Feb 20, 2024 at 3:23 PM Valentyn Tymofieiev via user < >>> user@beam.apache.org> wrote: >>> Hi Lydian, According to https://infra.apache.org/slack.html, invitations by link have been disabled. I submitted an invitation for your email address. Thanks, Valentyn On Tue, Feb 20, 2024 at 9:33 AM Lydian Lee wrote: > Hi, > > Can I get the invitation to join slack channel ? The ASF slack seems > required invitation to be able to join. Thanks >
