Contributor permission for Beam Jira tickets

Hi all, My name is Dmytro, I'm a Google engineer working on making interactive Beam more useful for Google's engineers (which should also hopefully help to use Dataflow runner interactively). Can someone please add me to Jira contributors, so that I could assign some issues to myself? My Jira

Re: Planning a freeze on website changes to merge new designs

On Mon, Jan 11, 2021 at 11:00 AM Robert Bradshaw wrote: > On Mon, Jan 11, 2021 at 10:38 AM Brian Hulette > wrote: > >> I spoke with Gris and Agnieszka about this on Friday. I should probably >> fill in the background a bit. >> >> The strategy we've adopted ro review the new designs so far is

Re: Usability regression using SDF Unbounded Source wrapper + DirectRunner

Hi, I proposed to make runner-issue checkpoint frequency configurable for a pipeline author here: https://docs.google.com/document/d/18jNLtTyyApx0N2ytp1ytOMmUPLouj2h08N3-4SyWGgQ/edit?usp=sharing. I believe it will also be helpful for the performance issue. Please feel free to drop any comments

Re: Have the Java examples gradle files broken?

Never mind - I think I found it. On Tue, Jan 12, 2021 at 12:31 PM Reuven Lax wrote: > If I try and run something from Java examples, e.g. the below: > > ./gradlew integrationTest -p examples/java --tests > org.apache.beam.examples.cookbook.BigQueryTornadoes ... > > > > I now get gradle errors

Accessing Custom Beam Metrics in Dataproc

Hi all, I'm currently in the process of adding some metrics to an existing pipeline that runs on Google Dataproc via Spark and I'm trying to determine how to access these metrics and eventually expose them to Stackdriver (to be used downstream in Grafana dashboards). The metrics themselves

Re: Can composite transforms have zero subtransforms?

Yes, a PTansform can have no sub-transforms, as long as it only returns its inputs. Updating the docs would be a good idea. On Tue, Jan 12, 2021 at 1:04 PM Brian Hulette wrote: > A recent bug with SqlTransform on Dataflow Runner V2 [1] revealed an > interesting ambiguity in the Beam model: it's

Re: [Input Needed] Updating the documentation left navigation (was:Website Revamp update & navigation for documentation page)

I see. Yes, I meant that things like (or some level) do not appear in the right nav. I see, if content is the next round I will save more comments. I think that once we have a style, including what levels are shown where, then it will be time to fix content issues. Kenn On Mon, Dec 21, 2020 at

Re: pulllicenses fails while building

FYI Emily fixed the bug with unintended license pulling. So bugs like the one Reuven originally reported should be bypassed by default. On Fri, Jan 8, 2021 at 5:01 PM Kyle Weaver wrote: > > $0.02 can we make it a separate target rather than an existing target > with flags? > > Something like

Can composite transforms have zero subtransforms?

A recent bug with SqlTransform on Dataflow Runner V2 [1] revealed an interesting ambiguity in the Beam model: it's not clear if a composite transform is allowed to have zero sub-transforms [2]. This may sound like an academic concern, but it can happen if a PTransform returns its own input, making

Have the Java examples gradle files broken?

If I try and run something from Java examples, e.g. the below: ./gradlew integrationTest -p examples/java --tests org.apache.beam.examples.cookbook.BigQueryTornadoes ... I now get gradle errors that look like this. Does anyone know if something changed? 2021-01-12T12:28:51.428-0800 [ERROR]

Re: Beam with Confluent Schema Registry and protobuf

Hi Alexey, The short answer is, I hope the code in my sample project makes its way back into Beam, but it may take some time. Hopefully until then people who are in a similar position have a demo of a workaround they can work with. The longer answer is: The history of this is a bit lengthy, let

Re: [Input needed] Capability Matrix Visual Redesign for extended version

Hi Kenn, thank you for your answers! We'll update the Capability Matrix UI according to your design related suggestions. It will be implemented in a way that allows you to rather easily change its text content, remove/add columns and rows etc. :) Kind regards, Agnieszka On Wed, Jan 6, 2021 at

Re: Beam with Confluent Schema Registry and protobuf

Hi Cristian, Great! Would you be interested to add this functionality into original ConfluentSchemaRegistryDeserializerProvider (along with Json support maybe) [1] ? Though, it seems that it will require an update of Confluent deps to at least version 5.5 [2] and it can be blocked by Beam