Re: Should we support VCF IO on Python 3?

Given that we don't support Python 2 anymore, it sounds like this is just broken code and we cannot expect anybody to be using it (after Beam 2.24.0). If so +1 for removing it from the codebase. If we decide to add it back with Python3 support, we should be able to refer to (working) 2.24.0

[ANNOUNCE] Beam 2.28.0 Released

The Apache Beam team is pleased to announce the release of version 2.28.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release

Re: Should we support VCF IO on Python 3?

Hi Yoshiki, If switching the code to a new version of VCF package is something easy to do, I would keep the code, but keep the dependency on vcf packages optional, since we know that this code is not in use. If you decide to try this route, https://issues.apache.org/jira/browse/BEAM-5628

Re: Java/Python/Proto mismatch: MergeStatus.ALREADY_MERGED vs InvalidWindows

On Mon, Feb 22, 2021 at 3:28 PM Kenneth Knowles wrote: > Good point that it is required to have a cross-language spec here. > > Yes, I think it is a property of the WindowFn but maybe also a property of > the pipeline as a whole. I've only really seen sessions, > sessions-with-some-limitation,

Re: Java/Python/Proto mismatch: MergeStatus.ALREADY_MERGED vs InvalidWindows

Good point that it is required to have a cross-language spec here. Yes, I think it is a property of the WindowFn but maybe also a property of the pipeline as a whole. I've only really seen sessions, sessions-with-some-limitation, and the wacky Nexmark WindowFns that merge everything for an

Re: Java/Python/Proto mismatch: MergeStatus.ALREADY_MERGED vs InvalidWindows

So, what we want to prohibit is stacked merging aggregations. (Open question: is that a property of the WindowFn, in which case some merging WindowFns could allow stacking, and some non-merging ones prohibit it, or is this really tied to merging itself?) In order to do this in a cross-language

Re: Do we need synchronized processing time? / What to do about "continuation triggers"?

I really wish that we had found the time to build sink triggers. Jan is right - specifying triggers up front and having them propagate down is confusing (it's also a bit confusing for Windows, but with Windows the propagation at least makes sense). The fact that users rarely have access to the

Re: Do we need synchronized processing time? / What to do about "continuation triggers"?

On Mon, Feb 22, 2021 at 11:51 AM Kenneth Knowles wrote: > I agree completely: Triggers control the output of the GBK. > > The issue is composite transforms, where there will be a GBK deep inside > some code and the user cannot adjust the triggering. > > What a user really wants is "sink triggers

[GitHub] [beam-site] chamikaramj merged pull request #611: Publish 2.28.0 release

chamikaramj merged pull request #611: URL: https://github.com/apache/beam-site/pull/611 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: Do we need synchronized processing time? / What to do about "continuation triggers"?

I agree completely: Triggers control the output of the GBK. The issue is composite transforms, where there will be a GBK deep inside some code and the user cannot adjust the triggering. What a user really wants is "sink triggers " [1], a purely

Re: [Vote] Publishing new website designs

+1. Let's merge. On Mon, Feb 22, 2021 at 11:28 AM Griselda Cuevas wrote: > Thanks Brian, this is a good plan! > > > On Mon, 22 Feb 2021 at 11:25, Brian Hulette wrote: > >> Since the only significant point of contention was the become a committer >> page I've reverted the changes to that page

Re: [Vote] Publishing new website designs

Thanks Brian, this is a good plan! On Mon, 22 Feb 2021 at 11:25, Brian Hulette wrote: > Since the only significant point of contention was the become a committer > page I've reverted the changes to that page (and the forked "attributes" > page), and I will go ahead and merge. I'll open a

Re: [Vote] Publishing new website designs

Since the only significant point of contention was the become a committer page I've reverted the changes to that page (and the forked "attributes" page), and I will go ahead and merge. I'll open a follow-up PR for the become a committer changes. Thanks everyone! Brian On Wed, Feb 17, 2021 at

Using Prometheus Client Metrics in Beam with FlinkRunner

Is anyone aware of a way to get Prometheus metrics to output via Beam (i.e. using them over the Beam abstractions)? I seem to be able to define them as expected, however I don’t see them being emitted with the rest of my Prometheus metrics: private class RouteEvent(): DoFn<...>() {

Beam Dependency Check Report (2021-02-22)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue chromedriver-binary 88.0.4324.96.0

Re: Do we need synchronized processing time? / What to do about "continuation triggers"?

The same holds true for pane accumulation mode.  Jan On 2/22/21 10:21 AM, Jan Lukavský wrote: Hi, I'm not sure if I got everything from this thread right, but from my point of view, triggers are property of GBK. They are property of neither windowing, nor PCollection, but relate solely to

Re: Do we need synchronized processing time? / What to do about "continuation triggers"?

Hi, I'm not sure if I got everything from this thread right, but from my point of view, triggers are property of GBK. They are property of neither windowing, nor PCollection, but relate solely to GBK. This can be seen from the fact, that unlike windowFn, triggers are completely ignored in

Apache Beam - Flink runner - FileIO.write issues in S3 writes after deploys

Hi, I am currently working on a Beam pipeline (2.23) (Flink runner - 1.8) where we read JSON events from Kafka and write the output in parquet format to S3. We write to S3 after every 10 min. We have observed that our pipeline sometimes stops writing to S3 after making minor non breaking code