Re: [VOTE] Release 2.10.0, release candidate #2

+1 (binding) Quickly tested on beam-samples. Regards JB On 05/02/2019 23:57, Kenneth Knowles wrote: > Hi everyone, > > Please review and vote on the release candidate #2 for the > version 2.10.0, as follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide

Re: [DISCUSS] Should File based IOs implement readAll() or just readFiles()

Since it seems we have consensus on deprecating both transforms I created BEAM-6605 Deprecate TextIO.readAll() and TextIO.ReadAll transform BEAM-6606 Deprecate AvroIO.readAll() and AvroIO.ReadAll transform Thanks everyone. On Fri, Feb 1, 2019 at 7:03 PM Chamikara Jayalath wrote: > > Python SDK

Re: [DISCUSS] Should File based IOs implement readAll() or just readFiles()

+1 Thanks for that Ismaël. Regards JB On 06/02/2019 11:24, Ismaël Mejía wrote: > Since it seems we have consensus on deprecating both transforms I created > > BEAM-6605 Deprecate TextIO.readAll() and TextIO.ReadAll transform > BEAM-6606 Deprecate AvroIO.readAll() and AvroIO.ReadAll transform >

Re: [VOTE] Release 2.10.0, release candidate #2

Hi,I checked Nexmark on both output size (functional regression detection) and run time (performance regression). The only thing I see is a performance regression on query7 (side input + fanout) in spark runner but this regression is there since the previous release cut.Indeed 2.9 was cut on

Re: [VOTE] Release 2.10.0, release candidate #2

+1. I verified the source artifacts look good, and tried the Python wheels. On Tue, Feb 5, 2019 at 11:57 PM Kenneth Knowles wrote: > > Hi everyone, > > Please review and vote on the release candidate #2 for the version 2.10.0, as > follows: > > [ ] +1, Approve the release > [ ] -1, Do not

Re: [VOTE] Release 2.10.0, release candidate #1

Hi, I just fixed both (one was not a bug but an error in test code) in this [1] PR[1] https://github.com/apache/beam/pull/7751 Etienne Le mardi 05 février 2019 à 17:37 +0100, Etienne Chauchot a écrit : > Hi guys, > I just found 2 bugs while replacing the mock in CassandraIO by a proper >

Re: 2.7.1 (LTS) release?

Having gone through the release process, I have a couple of git drawings to share. Currently the release process looks like this (you'll have to view in fixed width font if it is stripped by the mail manager). -X master \ ---Y-a--b---c-

Re: [VOTE] Release 2.10.0, release candidate #2

- Ran Flink WordCount with Quickstart guide - Ran release testing scripts for Flink Discovered a regression: https://jira.apache.org/jira/browse/BEAM-6608 If there is another blocker for the release, I'd would like to fix this for RC3. PR is already out. Thanks, Max On 06.02.19 11:24,

Resource usage exceeded: topics-per-project

Hi everyone, Our python pipelines failed with limit exceeded error : ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=1, maximum=1). Does anyone know if there were new

Re: [VOTE] Release 2.10.0, release candidate #2

Thank you. Here it is: https://github.com/apache/beam/pull/7753 On 06.02.19 18:30, Kenneth Knowles wrote: OK. Canceling this vote. Can you please simultaneously open a cherrypick so we can move it along at the same time? On Wed, Feb 6, 2019 at 9:25 AM Kenneth Knowles >

Re: Resource usage exceeded: topics-per-project

To clarify, PubsubIO does not clean up auto-created subscriptions, and SQL doesn't compensate for that. On Wed, Feb 6, 2019 at 8:45 AM Mikhail Gryzykhin wrote: > Thank you for quick response Andrew. > > I'll cleanup these. I'll keep the bug open and assign it to @Kenneth > Knowles who's

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

+1 to have a registry for images accessible to anyone. For snapshot images, I agree that gcr + apache-beam-testing project seems a good and easy way to start with. Łukasz wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): > +1 to have an official Beam released container image. > > Also I would

Re: Resource usage exceeded: topics-per-project

+Kenneth Knowles you're working on SQL recently, so might provide some info. I see a lot of topics of format rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721

Re: [DISCUSSION] UTests and embedded backends

Hi guys, I just submitted the PR: https://github.com/apache/beam/pull/7751. It contains refactorings, tests improvements/fixes and production code fixing. I wanted to give a little feedback because replacing the mock by a real instance allowed to - improve the tests: fix bad tests- add missing

Re: Resource usage exceeded: topics-per-project

SQL doesn't cleanup pubsub subscriptions. Feel free to delete those. Andrew On Wed, Feb 6, 2019, 8:01 AM Mikhail Gryzykhin wrote: > +Kenneth Knowles you're working on SQL recently, so > might provide some info. > > I see a lot of topics of format >

Re: Resource usage exceeded: topics-per-project

Thank you for quick response Andrew. I'll cleanup these. I'll keep the bug open and assign it to @Kenneth Knowles who's working on SQK for follow up: we need a way to automatically cleanup topics. Current suggestions: 1. Make SQL cleanup created topics 2. Cleanup topics created by SQL in tests

Re: Resource usage exceeded: topics-per-project

Ah, I jumped to the wrong vocabulary. There is no auto-created topic. That is part of TestPubsub and TestPubsubSignal. It should be cleaned up. The only use of those is in SQL ITs. But PubsubIO _should_ use it. Kenn On Wed, Feb 6, 2019 at 9:04 AM Kenneth Knowles wrote: > To clarify, PubsubIO

Re: Resource usage exceeded: topics-per-project

Minor UPD: As expected it fails most of our test jobs, since we use Pub/Subs in many tests. --Mikhail Have feedback ? On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin wrote: > Hi everyone, > > Our python pipelines failed with limit exceeded error >

Re: Resource usage exceeded: topics-per-project

Oops, I'm mixing up terms here. Topics != Subscriptions. We shouldn't be leaking topics. Andrew On Wed, Feb 6, 2019 at 9:04 AM Kenneth Knowles wrote: > To clarify, PubsubIO does not clean up auto-created subscriptions, and SQL > doesn't compensate for that. > > On Wed, Feb 6, 2019 at 8:45 AM

Re: [VOTE] Release 2.10.0, release candidate #2

Quick clarification: I linked to the wrong verification spreadsheet tab. The one for 2.10.0 is https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=467787719 On Wed, Feb 6, 2019 at 7:33 AM Maximilian Michels wrote: > - Ran Flink WordCount with Quickstart

Re: Beam Python streaming pipeline on Flink Runner

Thanks for your replies Robert and Cham. What I had in mind was a generic Wrapper that would easily allow users to use IO from Java. Such wrapper could start as an experimental feature and then, through URN versioning, become stable eventually. UDFs are needed, though they are a special

Re: [VOTE] Release 2.10.0, release candidate #2

OK. Canceling this vote. Can you please simultaneously open a cherrypick so we can move it along at the same time? On Wed, Feb 6, 2019 at 9:25 AM Kenneth Knowles wrote: > Quick clarification: I linked to the wrong verification spreadsheet tab. > The one for 2.10.0 is >

Re: [DISCUSS] (Forked thread) Beam issue triage & assignees

While we work with infra on this, let's remove the broken system and use tags. It is important that issues coming in are known to be untriaged, so instead of a "Needs Triage" label, we should use "triaged". So I will take these actions that everyone seems to agree on: - Remove default assignment

Re: [VOTE] Release 2.10.0, release candidate #3

+1 (binding) On Wed, Feb 6, 2019 at 2:28 PM Kenneth Knowles wrote: > Hi everyone, > > Please review and vote on the release candidate #3 for the version 2.10.0, > as follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete

Re: [DISCUSS] (Forked thread) Beam issue triage & assignees

I re-triaged most issues where the creation date != last update. I worked through everyone with more issues than myself (which I have triaged regularly) and a few people with a few fewer issues. I didn't look as closely at issues that were filed by the assignee. So if you filed a bunch of issues

Re: Another another new contributor! :)

Welcome! On Tue, 5 Feb 2019 at 23:34, Kenneth Knowles wrote: > Welcome Kyle! > > On Tue, Feb 5, 2019 at 4:34 AM Maximilian Michels wrote: > >> Welcome Kyle! Excited to see the Spark Runner moving towards portability! >> >> On 05.02.19 01:14, Connell O'Callaghan wrote: >> > Welcome Kyle! >> >