Look at the GroupIntoBatches[1] transform. It will buffer "batches" of size
X for you.
1:
https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/
On Thu, Oct 1, 2020 at 2:51 PM Vincent Marquez
wrote:
> the downstream consumer has these requirements.
>
> *~Vincent*
>
the downstream consumer has these requirements.
*~Vincent*
On Thu, Oct 1, 2020 at 2:29 PM Luke Cwik wrote:
> Why do you want to only emit X? (e.g. running out of memory in the runner)
>
> On Thu, Oct 1, 2020 at 2:08 PM Vincent Marquez
> wrote:
>
>> Hello all. If I want to 'throttle' the
Why do you want to only emit X? (e.g. running out of memory in the runner)
On Thu, Oct 1, 2020 at 2:08 PM Vincent Marquez
wrote:
> Hello all. If I want to 'throttle' the number of messages I pull off say,
> Kafka or some other queue, in order to make sure I only emit X amount per
> trigger, is
Hello all. If I want to 'throttle' the number of messages I pull off say,
Kafka or some other queue, in order to make sure I only emit X amount per
trigger, is there a way to do that and ensure that I get 'at least once'
delivery guarantees? If this isn't supported, would the better way be to
I would suggest trying FlinkRunner as it is a much more complete streaming
implementation.
SparkRunner has several key things that are missing that won't allow your
pipeline to function correctly.
If you're really invested in getting SparkRunner working though feel free
to contribute the necessary
This should be easily doable by adding an additional tag onto existing
docker images. I can help with that.
On Thu, Oct 1, 2020 at 10:28 AM Luke Cwik wrote:
> Can we copy the beam_java_sdk image to beam_java8_sdk for a few prior
> releases so people who are on an older release can migrate now
Can we copy the beam_java_sdk image to beam_java8_sdk for a few prior
releases so people who are on an older release can migrate now and not have
to remember to do it with 2.26?
On Tue, Sep 29, 2020 at 5:37 PM Emily Ye wrote:
> Starting with the release of 2.26.0, the Java 8 SDK container image
Welcome to the community. I have added you as a contributor.
Please take a look at the contribute guide[1].
1: https://beam.apache.org/contribute/
On Thu, Oct 1, 2020 at 9:49 AM George Pearman wrote:
> Hi
>
> My name is George. I have been developing Beam applications at LinkedIn
> for the
Hi
My name is George. I have been developing Beam applications at LinkedIn for
the past 1.5 years and I would like to start contributing back to Beam open
source.
Can someone please add me as a contributor for Beam's Jira issue tracker? I
would like to create/assign tickets for my work.
My
