Re: Builds Meeting this Thursday

Hi all, In case you may find this interesting / valuable: Airflow has configured their own machines for Github actions. Here's the PR https://github.com/apache/airflow/pull/13730 And here's the thread:

Re: Publish on Pubsub with ordering keys

Beam currently does not support this feature. Please see here for some context (with regard to Beam+Dataflow): https://medium.com/google-cloud/google-cloud-pub-sub-ordered-delivery-1e4181f60bc8 Thanks, Chamikara On Mon, Feb 8, 2021 at 5:13 PM Hemali Sutaria wrote: > > >

Re: [PROPOSAL] Preparing for Beam 2.28.0 release

Update: Cherry-picks are in. Validating the release branch here: https://github.com/apache/beam/pull/13941 Thanks, Cham On Wed, Feb 3, 2021 at 6:41 PM Chamikara Jayalath wrote: >

Re: [Vote] Publishing new website designs

@Ahmet Altay : Thank you for the suggestions to the Powered By page! I created two JIRA tickets for each of these. On Tue, Feb 9, 2021 at 11:34 AM Brian Hulette wrote: > > Optional point: does it make sense to maintain a link to the old Website > > from > the new one for some time (in case

Re: COVAR_POP aggregate function test for the ZetaSql dialec

Looks like the code that converts the parsed ZetaSQL to a Calcite logical expression doesn't currently support aggregate functions with multiple columns. See this TODO:

COVAR_POP aggregate function test for the ZetaSql dialec

Hi Devs, I am trying to test the COVAR_POP aggregate function for the ZetaSql dialect. I see https://github.com/apache/beam/blob/b74fcf7b30d956fb42830d652a57b265a1546973/sdks/[…]he/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java

Re: Unit Testing Kafka in Apache Beam

Ah this is getting trickier :) So in the Cloud PubSub tests that I'm referring to we don't use PAssert, we instead just inject some messages to one PubSub topic, and write some output to another PubSub topic that we can then verify. We're also not doing any verification of event time semantics.

Re: [Vote] Publishing new website designs

> Optional point: does it make sense to maintain a link to the old Website from the new one for some time (in case something gets broken in the new version or some information becomes unavailable) ? We could do this, but I think it would require duplicating all the source for the existing website

Re: Unit Testing Kafka in Apache Beam

Hey Brian, So that’s an approach that was thinking about last night and I’m sure it could work (I.e. running the pipeline and shortly after sending through messages). I’m assuming that as long as the pipeline is running, I should be able to apply a PAssert against it to verify that something

Re: Unit Testing Kafka in Apache Beam

Hi Rion, Can you run the pipeline asynchronously and inject messages after it has started? We use this approach for some tests against Cloud PubSub. Note if using the DirectRunner you need to set the blockOnRun pipeline option to False to do this. Brian On Mon, Feb 8, 2021 at 2:10 PM Rion