[ANNOUNCE] Beam 2.32.0 Released

The Apache Beam team is pleased to announce the release of version 2.32.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release

Re: [ANNOUNCE] New committer: Yichi Zhang

Congrats Yichi! On Thu, Apr 22, 2021 at 10:13 AM Yichi Zhang wrote: > Thanks everyone! It's my honor and I hope I can make more contributions in > the future! > > On Thu, Apr 22, 2021 at 10:11 AM Yichi Zhang wrote: > >> Thanks, Brian! >> >> On Thu, Apr 22, 2021 at 9:11 AM Brian Hulette >>

Re: Java Tests are failing on Github checks

> wrote: > >> Hi Ankur >> >> I've created a PR https://github.com/apache/beam/pull/14148 to see if >> this upgrade might help. >> >> Would be great to know if you can repro locally on Ubuntu. I'm on a Mac. >> >> thanks >> --shehzaad >>

Re: Java Tests are failing on Github checks

The :sdks:java:core:compileJava is only failing for ubuntu on error prone. I am trying to repro it locally and can try updating the error prone version to see if it helps. On Thu, Mar 4, 2021 at 3:26 PM Shehzaad Nakhoda wrote: > It appears this is an issue in errorprone:2.3.1 which has been

Re: [ANNOUNCE] New PMC Member: Chamikara Jayalath

Congrats Cham! On Thu, Jan 21, 2021 at 2:57 PM Ahmet Altay wrote: > Hi all, > > Please join me and the rest of Beam PMC in welcoming Chamikara Jayalath as > our > newest PMC member. > > Cham has been part of the Beam community from its early days and > contributed to the project in significant

Re: [PROPOSAL] Preparing for Beam 2.28.0 release

Thanks Cham! On Thu, Jan 14, 2021 at 12:26 PM Yichi Zhang wrote: > Thank you, Cham! > > On Wed, Jan 13, 2021 at 12:58 PM Ahmet Altay wrote: > >> Thank you Cham! >> >> On Wed, Jan 13, 2021 at 12:44 PM Rui Wang wrote: >> >>> Thanks Cham for working on this! >>> >>> >>> -Rui >>> >>> On Wed, Jan

Re: [ANNOUNCE] New PMC Member: Alexey Romanenko

Congratulations Alexey! On Tue, Jun 16, 2020 at 2:41 PM Thomas Weise wrote: > Congratulations! > > > On Tue, Jun 16, 2020 at 1:27 PM Valentyn Tymofieiev > wrote: > >> Congratulations! >> >> On Tue, Jun 16, 2020 at 11:41 AM Ahmet Altay wrote: >> >>> Congratulations! >>> >>> On Tue, Jun 16,

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

Thanks Valentyn! On Mon, Jun 15, 2020 at 12:41 PM Kyle Weaver wrote: > Sounds good, thanks Valentyn! > > On Mon, Jun 15, 2020 at 12:31 PM Valentyn Tymofieiev > wrote: > >> Hi all, >> >> According to the Beam release calendar [1], the next (2.23.0) release >> branch cut is scheduled for July 1.

Re: CVE Audit Plugin for Python

+1 for adding it to Python. We can explore more as to how we can surface the findings as a health signal. It will also be good to apply it to our old releases for users to be aware of. On Fri, May 29, 2020 at 11:20 AM Luke Cwik wrote: > Past work added an audit plugin for Java[1]. I reached

Re: How to submit PRs for dependant changes?

I would prefer c. I suppose you already have 4 separate changes in 4 separate local branches, Each of a,b and c will require merging locally. So there shouldn't be much of a difference in effort in these 3 options. c would be easiest to review and will preserve information in the most meaningful

Re: Default WindowFn for Unbounded source

Hi Amit, As you don't have any GroupByKey or trigger in your pipeline, you don't need to do allowed lateness. For unbounded source, Global window will never fire a trigger or emit GroupByKey. In the code you linked, a trigger is used which uses allowedLateness. Thanks, Ankur On Tue, Mar 31,

Re: Tests not getting triggered

Seems to be running now with some delay. On Thu, Mar 26, 2020 at 12:33 PM Luke Cwik wrote: > I saw upwards of a 20 min delay before anything was triggered on a couple > of PRs. > > On Thu, Mar 26, 2020 at 12:32 PM Ankur Goenka wrote: > >> Hi, >> >> I think

Tests not getting triggered

Hi, I think the tests for PRs are not getting triggered example: https://github.com/apache/beam/pull/10870 Can someone take a look. Thanks, Ankur

Re: [ANNOUNCE] New committer: Chad Dombrova

Congratulations Chad! On Mon, Feb 24, 2020 at 3:34 PM Ahmet Altay wrote: > Congratulations! > > On Mon, Feb 24, 2020 at 3:25 PM Sam Bourne wrote: > >> Nice one Chad. Your typing efforts are very welcomed. >> >> On Tue, Feb 25, 2020 at 10:16 AM Yichi Zhang wrote: >> >>> Congratulations, Chad!

Re: [RELEASE VOTE RESULT] Release 2.19.0, release candidate #1

Thanks Boyuan! A record has been set :) On Mon, Feb 3, 2020 at 5:05 PM Udi Meiri wrote: > Thank you Boyuan! > > On Mon, Feb 3, 2020 at 3:40 PM Ahmet Altay wrote: > >> On Mon, Feb 3, 2020 at 1:22 PM Thomas Weise wrote: >> >>> Impressive, probably the fastest/smoothest Beam release so far. >>>

Re: [ANNOUNCE] New committer: Hannah Jiang

Congrats Hannah! On Tue, Jan 28, 2020 at 7:30 PM Reza Rokni wrote: > Congratz! > > On Wed, 29 Jan 2020 at 09:52, Valentyn Tymofieiev > wrote: > >> Congratulations, Hannah! >> >> On Tue, Jan 28, 2020 at 5:46 PM Udi Meiri wrote: >> >>> Welcome and congrats Hannah! >>> >>> On Tue, Jan 28, 2020

Re: [ANNOUNCE] Beam 2.18.0 Released

Thanks Udi! On Tue, Jan 28, 2020 at 11:30 AM Yichi Zhang wrote: > Thanks Udi! > > On Tue, Jan 28, 2020 at 11:28 AM Hannah Jiang > wrote: > >> Thanks Udi! >> >> >> On Tue, Jan 28, 2020 at 11:09 AM Pablo Estrada >> wrote: >> >>> Thanks Udi! >>> >>> On Tue, Jan 28, 2020 at 11:08 AM Rui Wang

Re: Ordering of element timestamp change and window function

ction after the changed timestamp and before a trigger. Though, this can only be validated at the runtime in python. 4. Revalidating the window after changing the timestamp. Also provide additional methods to explicitly change the timestamp and window in oneshot. 5. etc > Kenn > >

Re: Ordering of element timestamp change and window function

the full windowed value (accepted without checking...), a > timestamped value (in which case the window will be computed), or a > plain old element (in which case the window and timestamp will be > computed (really, propagated)). > > On Wed, Jan 15, 2020 at 3:51 PM Ankur Goenka wrote:

Re: Ordering of element timestamp change and window function

dow. > > On Tue, Jan 14, 2020 at 7:30 PM Ankur Goenka wrote: > >> Hi, >> >> I am not sure about the effect of the order of element timestamp change >> and window association has on a group by key. >> More specifically, what would be the behavior if we apply

Ordering of element timestamp change and window function

Hi, I am not sure about the effect of the order of element timestamp change and window association has on a group by key. More specifically, what would be the behavior if we apply window -> change element timestamp -> Group By key. I think we should always apply window function after changing the

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

Also curious to know if apache provide any infra support fro projects under Apache umbrella and any quota limits they might have. On Fri, Jan 10, 2020, 2:26 PM Robert Bradshaw wrote: > One downside is that, unlike many of these projects, we release a > dozen or so containers. Is there exactly

Re: [ANNOUNCE] Beam 2.17.0 Released!

Thanks for persistent and powering through all the issues. On Fri, Jan 10, 2020 at 10:23 AM Maximilian Michels wrote: > At last :) Thank you for making it happen Mikhail! Also thanks to > everyone else who tested the release candidate. > > Cheers, > Max > > On 10.01.20 19:01, Mikhail Gryzykhin

Re: Cleaning up SDK docker image tagging

>> For the released version of SDKs, the default tag will be version number. (ex: 2.17.0) +1 >> For the unreleased version of SDKs, the default tag will be version number + '.dev'. (ex: 2.18.0.dev) Shall we ALSO tag the image with git commit version for local build to keep track of obsolete

Re: Java PortableRunner GBK load test fails

The connection closing is a red herring as that error gets printed when SDK Harness dies. More logs/jenkins link would be useful to understand what going on with the pipeline. On Tue, Dec 17, 2019 at 5:53 AM Michał Walenia wrote: > Hi there, > I'm trying to add a Jenkins job for a load test of

Re: Pipeline parameters for running jobs in a cluster

Hi Matthew, For 1: Beam does not compute the right configuration for the pipeline so its recommended to tune it manually as it's done in regular Spark jobs. For 2: The recommendation is same as that for a regular Spark job. Thanks, Ankur On Tue, Dec 10, 2019 at 2:46 PM Matthew K. wrote: >

Re: Version Beam Website Documentation

in mind that > would benefit from versioning? Are there specific cases you see where the > current website would be confusing for someone using a Beam SDK that's a > few versions old? > > On Tue, Dec 3, 2019 at 6:46 PM Ankur Goenka wrote: > >> Hi, >> >> We are constant

Version Beam Website Documentation

Hi, We are constantly adding features to Beam which makes each new Beam version more feature rich and compelling. This also means that the old Beam released don't have the new features and might have different ways to do certain things. (I might be wrong here) - Our Beam website only publish a

Re: [ANNOUNCE] New committer: Daniel Oliveira

Congrats Daniel! On Mon, Nov 25, 2019 at 10:02 PM Tanay Tummalapalli wrote: > Congratulations! > > On Mon, Nov 25, 2019 at 11:12 PM Mark Liu wrote: > >> Congratulations, Daniel! >> >> On Mon, Nov 25, 2019 at 9:31 AM Ahmet Altay wrote: >> >>> Congratulations, Daniel! >>> >>> On Sat, Nov 23,

Re: [ANNOUNCE] New committer: Brian Hulette

Congrats Brian! On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský wrote: > Congrats Brian! > On 11/15/19 9:58 AM, Reza Rokni wrote: > > Great news! > > On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov wrote: > >> Congratulations! >> >> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev >> wrote: >> >>>

Re: [PROPOSAL] Preparing for Beam 2.17.0 release

Thanks Mikhail! +1 On Tue, Oct 15, 2019 at 11:38 AM Mikhail Gryzykhin wrote: > Hi all, > > Beam 2.17 release branch cut is scheduled on Oct 23 according to the > release calendar [1]. I would like to volunteer myself to do this release. > The plan is to cut the branch on that date, and

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019 at 4:27 PM Ruoyun Huang wrote: > +1 > > On Tue, Oct 1, 2019 at 3:52 PM Rui Wang wrote: > >> +1 >> >> I needed to use https://python3statement.org to access the website BTW >> (https, not http). >> >> >> -Rui >> >> On Tue, Oct 1, 2019 at 3:29 PM Cam Mach wrote: >> >>> +1

Re: [ANNOUNCE] New committer: Alan Myrvold

Congratulations Alan! On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang wrote: > Congrats, Alan! > > On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu wrote: > >> Congrats, Alan! >> >> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang >> wrote: >> >>> Congrats Alan! >>> >>> On Fri, Sep 27, 2019 at 9:57 AM

Re: Collecting feedback for Beam usage

of "Oh no, your Flink pipeline crashed! Check Jira/file a bug/ask the > mailing list." > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > > > On Fri, Sep 20, 2019 at 4:14 PM Ankur Goenka wrote: > >> Hi, >> >> At the moment

Collecting feedback for Beam usage

Hi, At the moment we don't really have a good way to collect any usage statistics for Apache Beam. Like runner used etc. As many of the users don't really have a way to report their usecase. How about if we create a feedback page where users can add their pipeline details and usecase. Also, we

Re: Improve container support

Please ignore the previous email. I was looking at the older document in the mail thread. On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka wrote: > I think sdk in the name is obsolete as they are all under sdks name space. > > On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang > wrote:

Re: Improve container support

I think sdk in the name is obsolete as they are all under sdks name space. On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang wrote: > Hi Team > > Thanks for all the comments about beam containers. > After considering various opinions and investigating gcr and docker hub, > we decided to push images

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

Congratulations Valentyn! On Mon, Aug 26, 2019, 5:02 PM Yifan Zou wrote: > Congratulations, Valentyn! Well deserved! > > On Mon, Aug 26, 2019 at 3:31 PM Aizhamal Nurmamat kyzy < > aizha...@google.com> wrote: > >> Congratulations! and thank you for your contributions, Valentyn! >> >> On Mon, Aug

Re: Dataflow worker overview graphs

Thanks Mikhail. This is really useful. Do you also have something similar for Streaming use case. More specifically for Portable (fn_api) based streaming pipelines. On Thu, Aug 8, 2019 at 2:08 PM Mikhail Gryzykhin wrote: > Hello everybody, > > Just wanted to share that I have found some graphs

Re: [ANNOUNCE] New committer: Kyle Weaver

Congratulations Kyle! On Tue, Aug 6, 2019 at 9:35 AM Ahmet Altay wrote: > Hi, > > Please join me and the rest of the Beam PMC in welcoming a new committer: Kyle > Weaver. > > Kyle has been contributing to Beam for a while now. And in that time > period Kyle got the portable spark runner feature

Re: [ANNOUNCE] New committer: Rui Wang

Congratulations Rui! Well deserved  On Tue, Aug 6, 2019 at 9:35 AM Ahmet Altay wrote: > Hi, > > Please join me and the rest of the Beam PMC in welcoming a new committer: Rui > Wang. > > Rui has been an active contributor since May 2018. Rui has been very > active in Beam SQL [1] and continues

Re: [ANNOUNCE] New committer: Jan Lukavský

Congratulations Jan! On Wed, Jul 31, 2019, 1:23 AM David Morávek wrote: > Congratulations Jan, well deserved! ;) > > D. > > On Wed, Jul 31, 2019 at 10:17 AM Ryan Skraba wrote: > >> Congratulations Jan! >> >> On Wed, Jul 31, 2019 at 10:10 AM Ismaël Mejía wrote: >> > >> > Hi, >> > >> > Please

Re: Docker Run Options in SDK Container

Thanks for summarizing the discussion. A few comments inline below: On Mon, Jul 15, 2019 at 5:28 PM Sam Bourne wrote: > Hello Beam devs, > > I’ve opened a PR (https://github.com/apache/beam/pull/8982) to support > passing options/flags to the docker run command executed as part of the >

Re: [ANNOUNCE] New committer: Robert Burke

Congratulations Robert! Go GO! On Tue, Jul 16, 2019 at 10:34 AM Rui Wang wrote: > Congrats! > > > -Rui > > On Tue, Jul 16, 2019 at 10:32 AM Udi Meiri wrote: > >> Congrats Robert B.! >> >> On Tue, Jul 16, 2019 at 10:23 AM Ahmet Altay wrote: >> >>> Hi, >>> >>> Please join me and the rest of

Re: [ANNOUNCE] New committer: Mikhail Gryzykhin

Congrats Mikhail! On Fri, Jun 21, 2019 at 11:55 AM Tanay Tummalapalli wrote: > Congratulations! > > On Fri, Jun 21, 2019 at 10:35 PM Rui Wang wrote: > >> Congrats! >> >> >> -Rui >> >> On Fri, Jun 21, 2019 at 9:58 AM Robin Qiu wrote: >> >>> Congrats, Mikhail! >>> >>> On Fri, Jun 21, 2019 at

Re: Contributor Registration

Welcome Matt! On Thu, Jun 20, 2019 at 12:25 PM Gleb Kanterov wrote: > Welcome Matt! > > On Thu, Jun 20, 2019 at 11:09 AM Aizhamal Nurmamat kyzy < > aizha...@google.com> wrote: > >> Welcome Matt! >> >> On Thu, Jun 20, 2019 at 11:06 AM Robert Bradshaw >> wrote: >> >>> Welcome! I added you to the

Re: [PROPOSAL] Preparing for Beam 2.14.0 release

+1 On Thu, Jun 6, 2019, 9:13 AM Ahmet Altay wrote: > +1, thank you for keeping the cadence. > > On Thu, Jun 6, 2019 at 9:04 AM Anton Kedin wrote: > >> Hello Beam community! >> >> Beam 2.14 release branch cut date is June 19 according to the release >> calendar [1]. I would like to volunteer

Re: [VOTE] Release 2.13.0, release candidate #2

quests to >>>> have a JIRA filed against that release and marked as blockers? >>>> > >>>> > >>>> > I agree with this and with what Ankur said. Release blockers should >>>> be more explicit and should come with a reason. After voting

Re: [VOTE] Release 2.13.0, release candidate #2

t we ask all cherry pick requests to > have a JIRA filed against that release and marked as blockers? > > On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka wrote: > >> That makes sense. >> I would also like to add that the corresponding PR should be added to an >> open blocki

Re: [VOTE] Release 2.13.0, release candidate #2

; I would actually suggest that the following search needs to be triaged to > zero before cutting an RC: > https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0 > . > > On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka wrote: > >> Sorry, I mis

Re: [VOTE] Release 2.13.0, release candidate #2

this branch. > > Thanks, > Thomas > > > On Tue, Jun 4, 2019 at 1:59 AM Maximilian Michels wrote: > >> The summary is not correct. Binding votes (in order): >> >> Ahmet Altay >> Robert Bradshaw >> Maximilian Michels >> Jean-Baptiste Onofré &g

Re: [VOTE] Release 2.13.0, release candidate #2

ark java and SQL perfkit graphs, no obvious > regressions over the previous release. > > On Mon, Jun 3, 2019 at 1:15 PM Lukasz Cwik wrote: > >> Thanks for the clarification. >> >> On Mon, Jun 3, 2019 at 11:40 AM Ankur Goenka wrote: >> >>> Yes,

Re: [VOTE] Release 2.13.0, release candidate #2

Yes, i meant i will close the voting at 5pm and start the release process. On Mon, Jun 3, 2019, 10:59 AM Lukasz Cwik wrote: > Ankur, did you mean to say your going to close the vote today at 5pm? (and > then complete the release afterwards) > > On Mon, Jun 3, 2019 at 10:54 AM

Re: [VOTE] Release 2.13.0, release candidate #2

locker for this release and am voting +1. > > On Mon, Jun 3, 2019 at 7:13 AM Jean-Baptiste Onofré > wrote: > >> +1 (binding) >> >> Quickly tested on beam-samples. >> >> Regards >> JB >> >> On 31/05/2019 04:52, Ankur Goenka wrote: >> >

Re: [DISCUSS] Cookbooks for users with knowledge in other frameworks

+1 for the proposal. Compatibility Matrix can be a good place to show case parity between different runners. Do you think we should write 2 way examples [Spark, Flink, ..]<=>Beam? On Sat, Jun 1, 2019 at 4:31 PM Reza Rokni

Re: [VOTE] Release 2.13.0, release candidate #2

t[1] suggests that I was using a correct one) >> 2.10.0 and higher fail due to the missing gs-collections dependency. >> >> Manu, could you help figure out what is going on? >> >> 1: >> https://github.com/apache/beam/blob/2d3bcdc542536037c3e657a8b00ebc222487476b/r

Re: 1 Million Lines of Code (1 MLOC)

Thanks for sharing. This is really interesting metrics. One use I can see is to track LOC vs Comments to make sure that we keep up with the practice of writing maintainable code. On Fri, May 31, 2019 at 3:04 PM Ismaël Mejía wrote: > I was checking some metrics in our codebase and found by

[VOTE] Release 2.13.0, release candidate #2

Hi everyone, Please review and vote on the release candidate #2 for the version 2.13.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

gt;> not necessarily holding back. >> >> On Mon, May 27, 2019 at 7:42 PM Ankur Goenka wrote: >> > >> > +1 >> > We can release the images with 2.13 but we should not block 2.13 >> release for this. >> > >> > On Mon, May 27, 2019, 8:

Re: [VOTE] Release 2.13.0, release candidate #1

>> >>> -1. >>> I would like us to fix >>> https://issues.apache.org/jira/browse/BEAM-7439 for 2.13.0. It is a >>> regression that happened in 2.12.0, but was not caught by existing tests. >>> >>> Thanks, >>> Valentyn >

Re: [VOTE] Release 2.13.0, release candidate #1

Open cherry pick PRs for spark runner https://github.com/apache/beam/pull/8705 https://github.com/apache/beam/pull/8706 On Tue, May 28, 2019 at 3:42 PM Valentyn Tymofieiev wrote: > Yes, looking into that. > > On Tue, May 28, 2019 at 3:37 PM Ankur Goenka wrote: > >> Valent

Re: [VOTE] Release 2.13.0, release candidate #1

Hi All, In the meanwhile Please validate RC1 to catch anyother issues. Thanks, Ankur On Tue, May 28, 2019 at 3:37 PM Ankur Goenka wrote: > Valentyn, Can you please send the cherry pick PR for > https://issues.apache.org/jira/browse/BEAM-7439 > > On Tue, May 28, 2019 at 3:04 PM

Re: [VOTE] Release 2.13.0, release candidate #1

Valentyn, Can you please send the cherry pick PR for https://issues.apache.org/jira/browse/BEAM-7439 On Tue, May 28, 2019 at 3:04 PM Ankur Goenka wrote: > Sure, I will cherry pick those PRs. > > On Tue, May 28, 2019 at 2:19 PM Kyle Weaver wrote: > >> Hi Ankur, >> >&g

Re: [VOTE] Release 2.13.0, release candidate #1

run without errors on the 2.13.0 Spark runner ( > https://github.com/tensorflow/tfx/pull/84). > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > | +1650203 > > > On Tue, May 28, 2019 at 11:53 AM Ankur Goenka wrote: > >> Thanks for the validatio

Re: [VOTE] Release 2.13.0, release candidate #1

r reporting this issue. >>> >>> On Tue, May 28, 2019, 9:19 AM Valentyn Tymofieiev >>> wrote: >>> >>>> -1. >>>> I would like us to fix >>>> https://issues.apache.org/jira/browse/BEAM-7439 for 2.13.0. It is a >>>>

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

ramework. >> > >>> >> > >>> Mark >> > >>> >> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold >> wrote: >> > >>>> >> > >>>> +1 This would be great. gcr.io seems like a

[VOTE] Release 2.13.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.13.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: Enable security for data channels in portability

this secret to authenticate the worker (by using gRPC's interceptor for >>>>this customized auth) >>>>3. By having the 2 steps above, we *no longer need transport layer >>>>security *(SSL/TLS). So we abandon our initial plan to enable >>>&

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congratulations Pablo! On Wed, May 15, 2019, 12:21 PM Ruoyun Huang wrote: > Congratulations, Pablo! > > *From: *Charles Chen > *Date: *Wed, May 15, 2019 at 11:04 AM > *To: *dev > > Congrats Pablo and thank you for your contributions! >> >> On Wed, May 15, 2019, 10:53 AM Valentyn Tymofieiev >>

Re: All validates runner tests seems to be broken.

; Kenn is tracking a different issue related to publishing being broken > in BEAM-7302 which he has a fix for in > https://github.com/apache/beam/pull/8577 > > I suspect there will be a few of these gotchas. > > > *From: *Ankur Goenka > *Date: *Tue, May 14, 2019 at 1:43 PM >

Re: All validates runner tests seems to be broken.

> > Andrew > > *From: *Alan Myrvold > *Date: *Tue, May 14, 2019 at 1:59 PM > *To: * > > That is a typo added in https://github.com/apache/beam/pull/8194 >> >> https://github.com/apache/beam/commit/1e7ea0da5073566c3fa26dbc1105105fbe6043ae#diff-9591f0d06e82e71168

Re: All validates runner tests seems to be broken.

; after they all start on head again to >>> restore the config. >>> >>> Andrew >>> >>> *From: *Alan Myrvold >>> *Date: *Tue, May 14, 2019 at 1:59 PM >>> *To: * >>> >>> That is a typo added in https://github.com/apache/beam/pull/819

All validates runner tests seems to be broken.

Hi, Following tests seems to be broken because of "Project 'unners' not found in root project 'beam'." The command getting executed on Jenkins is gradlew --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g :unners:samza:validatesRunner causing the failure. The same

Re: Do we maintain offline artifact version in javadocs sdks/java/javadoc/build.gradle

gt;>> *To: *dev >>> >>> What is the difference between the two files you are referring to? >>>> >>>> Note that sdks/java/javadoc/build.gradle is meant to produce one giant >>>> javadoc across many modules that users would be interested

Re: Cassandra and hadoop test broken on master and in previous releases

ode) java version "1.8.0_202" Java(TM) SE Runtime Environment (build 1.8.0_202-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode) Thanks for validating the tests. Thanks, Ankur *From: *Ankur Goenka *Date: *Sat, May 11, 2019 at 7:21 PM *To: *dev Thanks for the quick va

Re: Cassandra and hadoop test broken on master and in previous releases

ild 1.8.0_172-b11) > Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) > > Regarding your Gradle scan, the JVM is crashing. > > Do you have a chance to try another JDK ? > > I will try with OpenJDK 1.8.0_181. > > Regards > JB > > On 11/05/2019 01:34, A

Do we maintain offline artifact version in javadocs sdks/java/javadoc/build.gradle

Hi, I see that the sdks/java/javadoc/build.gradle is not in sync with org/apache/beam/gradle/BeamModulePlugin.groovy . I wanted to check if we are maintaining or not based on that we can either remove or update sdks/java/javadoc/build.gradle. Thanks, Ankur

Cassandra and hadoop test broken on master and in previous releases

Hi, Cassandra and Hadoop tests for targets :beam-sdks-java-io-cassandra:test :beam-sdks-java-io-hadoop-format:test are failing at master and in 2.12.0 release with jvm crash. Gradle Scan: https://gradle.com/s/rhseoqeouup6e Any help on the debugging failure will be useful. Thanks, Ankur

Re: [PROPOSAL] Preparing for Beam 2.13.0 release

Branch for 2.13 is cut at revision number a0953215e08d756eb64c23c1916d483ba6e73945 Branch location: https://github.com/apache/beam/tree/release-2.13.0 Thanks, Ankur *From: *Ankur Goenka *Date: *Mon, May 6, 2019 at 11:27 AM *To: *dev Gentle reminder. > > We will be cutting the 2.13.0 bran

Re: Better naming for runner specific options

Having namespaces for option makes sense. I think, along with a help command to print all the options given the runner name will be useful. As for the scope of name spacing, I think that assigning a logical name space gives more flexibility around how and where we declare options. It also make

Re: [PROPOSAL] Preparing for Beam 2.13.0 release

Gentle reminder. We will be cutting the 2.13.0 branch day after tomorrow ( May 8th ). Please mark the release blockers as fixed and Fix Version as 2.13.0. Thanks, Ankur On Fri, Apr 26, 2019 at 2:28 PM Ankur Goenka wrote: > Link in the download link page will be useful. > > Add

Kotlin iterator error

Hi, A beam user on stackoverflow has posted issue while using kotlin sdk. https://stackoverflow.com/questions/55908999/kotlin-iterable-not-supported-in-apache-beam/55911859#55911859 I am not very familiar with kotlin so can someone please take a look. Thanks, Ankur

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congratulations Udi! On Fri, May 3, 2019 at 3:00 PM Connell O'Callaghan wrote: > Well done Udi!!! Congratulations and thank you for your contributions!!! > > Kenn thank you for sharing!!! > > On Fri, May 3, 2019 at 2:49 PM Yifan Zou wrote: > >> Thanks Udi and congratulations! >> >> On Fri, May

Re: Congrats to Beam's first 6 Google Open Source Peer Bonus recipients!

Congratulations and thank you for making Beam awesome! *From: *Chamikara Jayalath *Date: *Thu, May 2, 2019, 4:03 PM *To: *dev Congratulations! > > On Thu, May 2, 2019 at 10:28 AM Udi Meiri wrote: > >> Congrats everyone! >> >> On Thu, May 2, 2019 at 9:55 AM Ahmet Altay wrote: >> >>>

Re: Structured streaming based spark runner.

Exciting! Thanks Etienne for sharing the design and progress. On Tue, Apr 30, 2019 at 10:11 AM Etienne Chauchot wrote: > Hi guys, > As part of the ongoing work on spark runner POC based on structured > streaming framework, I sketched up a design doc (1) to share context and > design principles.

Re: Enable security for data channels in portability

In an offline chat with Hai, It seem useful for users to be able to provide custom authentication like a secret which can be distributed out of band by the infrastructure and can be provided via file system, rpc to another service etc. gRPC already has some mechanism for standard and custom

Re: [PROPOSAL] Preparing for Beam 2.13.0 release

Ankur, do you have the equivalent link that I can use to subscribe to >>>> that calendar via google calendars? >>>> The link seems to work only to see the calendar in a webpage. >>>> >>>> Thanks. >>>> >>>> On Fri, Apr 26, 2019 a

Re: [PROPOSAL] Preparing for Beam 2.13.0 release

Correction, The planned cut date is May 8th. On Thu, Apr 25, 2019 at 4:24 PM Ankur Goenka wrote: > Hello Beam community! > > Beam 2.13 release branch cut date is April 8th according to the release > calendar [1]. I would like to volunteer myself to do this release. I intend > to

[PROPOSAL] Preparing for Beam 2.13.0 release

Hello Beam community! Beam 2.13 release branch cut date is April 8th according to the release calendar [1]. I would like to volunteer myself to do this release. I intend to cut the branch as planned on April 8th and cherrypick fixes if needed. If you have releasing blocking issues for 2.13

Re: Integration of python/portable runner tests for Samza runner

Hi Daniel, We use flinkCompatibilityMatrix [1] to check the Flink compatibility with python. This is python equivalent to validatesRunner tests in java for portable runners. I think we can reuse it for Samza Portable runner with minor refactoring. [1]

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations Yifan! On Mon, Apr 22, 2019 at 11:42 AM Thomas Weise wrote: > Congrats Yifan! > > > On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels > wrote: > >> Congrats! Great work. >> >> -Max >> >> On 22.04.19 19:00, Rui Wang wrote: >> > Congratulations! Thanks for your contribution!!

Re: Artifact staging in cross-language pipelines

ansform can be configured to not perform these > functions). > > > > > > > > > To clarify, I'm only talking about transform expansion here. > > Many IO > > > transforms need read/write access to remote services at run >

Re: New IOIT Dashboards

This looks great! Which runner are we using for the pipeline? On Fri, Apr 19, 2019 at 12:03 PM Kenneth Knowles wrote: > Very cool! I assume times are all in seconds? > > On Fri, Apr 19, 2019 at 6:26 AM Łukasz Gajowy wrote: > >> Hi, >> >> just wanted to announce that we improved the way we

Re: Artifact staging in cross-language pipelines

I agree that the Expansion service knows about the artifacts required for a cross language transform and having a prepackage folder/Zip for transforms based on language makes sense. One think to note here is that expansion service might not have the same access privilege as the pipeline author

Re: [EXT] Re: [DOC] Portable Spark Runner

Thanks for sharing. This looks great! On Mon, Apr 15, 2019 at 2:54 PM Kenneth Knowles wrote: > Great. Thanks for sharing! > > On Mon, Apr 15, 2019 at 2:38 PM Lei Xu wrote: > >> This is super nice! Really look forward to use this. >> >> On Mon, Apr 15, 2019 at 2:34 PM Thomas Weise wrote: >>

Re: [review?] WordCount in Kotlin

Best > -P. > > On Fri, Apr 12, 2019, 12:22 PM Ankur Goenka wrote: > >> Thanks Pablo and Harshit. >> >> Just a quick reminder, please squash the "fixup" sort of commits in the >> PR based on the prior discussion on t

Re: [review?] WordCount in Kotlin

Thanks Pablo and Harshit. Just a quick reminder, please squash the "fixup" sort of commits in the PR based on the prior discussion on the mailing list https://lists.apache.org/thread.html/6d922820d6fc352479f88e5c8737f2c8893ddb706a1e578b50d28948@%3Cdev.beam.apache.org%3E On Fri, Apr 12, 2019 at

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

Congrats Boyuan! On Thu, Apr 11, 2019 at 4:52 PM Mark Liu wrote: > Congrats Boyuan! > > On Thu, Apr 11, 2019 at 9:53 AM Alexey Romanenko > wrote: > >> > since early 2018 >> > 100+ pull requests >> >> Wow, this is impressive! Great job, congrats! >> >> > On 11 Apr 2019, at 15:08, Maximilian

Re: [DISCUSS] Backwards compatibility of @Experimental features

I think a release version with Experimental flag makes sense. In addition, I think many of our user start to rely on experimental features because they are not even aware that these features are experimental and its really hard to find the experimental features used without giving a good look at

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations Mark! On Mon, Mar 25, 2019 at 12:04 PM Jason Kuster wrote: > Wonderful, congrats Mark! > > On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold wrote: > >> congratulations, Mark!!! >> >> On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang wrote: >> >>> Congratulations Mark! >>> >>> On Mon,

  1   2   >