[ANNOUNCE] Beam Java 8 image rename starting from 2.26.0 (to apache/beam_java8_sdk)

Starting with the release of 2.26.0, the Java 8 SDK container image (currently apache/beam_java_sdk ) will be released under a new name, *apache/beam_java8_sdk.* This is in anticipation of

Re: Behavior change for Gradle build target

A bisect found [1] to be the culprit, it upgrades the shadow plugin. I suspect it's related to this change in shadow 4.0.4 [2], but I'm not sure: > When using shadow, application, and maven plugins together, remove shadowDistZip and shadowDistTar from > configurations.archives so they are not

Re: Behavior change for Gradle build target

I think that output is still incorrect for :sdks:java:io:expansion-service, there should be both shaded and unshaded in ./sdks/java/io/expansion-service/build/libs On Tue, Sep 29, 2020 at 1:08 PM Tyson Hamilton wrote: > Or, possibly that PR already fixed the issue. Could you sync, retry again?

Re: Behavior change for Gradle build target

Or, possibly that PR already fixed the issue. Could you sync, retry again? Here is what I found after running the gradle task you mentioned: ttysonjh@tysonjh:~/Development/beam$ find . -name "*expansion-service*SNAPSHOT*"

Re: Behavior change for Gradle build target

Hm. It sounds like it is possibly related to the gradle upgrade to 6? I had a similar issue that I fixed for the Java nightly snapshot build in PR#12947 (https://github.com/apache/beam/pull/12947). It may require similar changes to the gradle.build file to generate the jar. On Tue, Sep 29, 2020

Support streaming side-inputs in the Spark runner

Hi Team, I have a streaming pipeline (built using Apache Beam with Spark Runner)which consumes events tagged with timestamps from Unbounded source (Kinesis Stream) and batch them into FixedWindows of 5 mins each and then, write all events in a window into a single / multiple files based on

Support streaming side-inputs in the Spark runner

Hi Team, I have a streaming pipeline (built using Apache Beam with Spark Runner)which consumes events tagged with timestamps from Unbounded source (Kinesis Stream) and batch them into FixedWindows of 5 mins each and then, write all events in a window into a single / multiple files based on

Re: How to write a Python wrapper for MQTT io

Thanks for the explanation, Chamikara. If I will run into BEAM-6868 then perhaps I should wait on implementing this... I don't want to put a lot of effort in only to have to stop and wait for 6868 to be completed. Is it currently being worked on?

Re: How to write a Python wrapper for MQTT io

We are still working on adding documentation on authoring cross-language transforms. Hopefully it will happen in the next quarter. For now, you'll have to follow existing examples and guidelines Brian provided above and follow existing examples. You might run into

Re: How to write a Python wrapper for MQTT io

Hi Brian, Thanks for your quick reply! It sounds like a lot of work, but your instructions are nice and concise, which I appreciate. Before I try to tackle this, I want to make sure that the issues related to xlang Kafka via Flink won't get in the way of implementing and using this. For example,