Re: [DISCUSS] Integrate Google Cloud AI functionalities

If it's in Java also be careful to align with the current google cloud IO's, certainly it's dependencies. The google IO's are not depending on the the newest client libraries and that's something we're sometimes struggling with when we depend on our own client libraries. So make sure to align

Re: [DISCUSS] Python static type checkers

On Mon, Jan 13, 2020 at 6:43 PM Chad Dombrova wrote: > > I agree with focusing one mypy for now, but I would propose soon after, >> or in parallel if it will be different folks, to work on pytype and enable >> it as a first class citizen similar to mypy. If there will be a large delta >> between

Ordering of element timestamp change and window function

Hi, I am not sure about the effect of the order of element timestamp change and window association has on a group by key. More specifically, what would be the behavior if we apply window -> change element timestamp -> Group By key. I think we should always apply window function after changing the

Re: Quota limitation for Java tests

Hi Beam committers, I encountered a similar problem today for "Run Dataflow ValidatesRunner": Dataflow quota error for jobs-per-project quota. Project apache-beam-testing is running 303 jobs.

Re: Jenkins jobs not running for my PR 10438

Done. +Kenneth Knowles , any updates from INFRA on this? On Tue, Jan 14, 2020 at 6:43 PM Tomo Suzuki wrote: > It hit Dataflow quota error again. Can somebody run > Run Dataflow ValidatesRunner > for https://github.com/apache/beam/pull/10554 ? > > On Tue, Jan 14, 2020 at 12:14 PM Tomo Suzuki

Re: Jenkins jobs not running for my PR 10438

It hit Dataflow quota error again. Can somebody run Run Dataflow ValidatesRunner for https://github.com/apache/beam/pull/10554 ? On Tue, Jan 14, 2020 at 12:14 PM Tomo Suzuki wrote: > > Valentyn, thank you. > > On Tue, Jan 14, 2020 at 12:05 PM Valentyn Tymofieiev > wrote: > > > > Done. If tests

Re: [VOTE] Release 2.18.0, release candidate #1

Here my second take: Hi everyone, Please review and vote on the release candidate #1 for the version 2.18.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: *

Re: [VOTE] Release 2.18.0, release candidate #1

Please don't do any Dataflow-based verifications yet, because we'll have to redo them once new Dataflow containers are built. On Tue, Jan 14, 2020 at 6:27 PM Ahmet Altay wrote: > I verified python 2 quickstarts with batch and streaming pipelines, wheel > files, and reviewed changes to the

Re: [VOTE] Release 2.18.0, release candidate #1

I verified python 2 quickstarts with batch and streaming pipelines, wheel files, and reviewed changes to the blog/website. Udi, could you send an updated version of the voting text with TODOs, template pieces removed? We can discuss changes to the template separately. My vote is +1 pending an

Re: Jenkins job execution policy

Thanks for working on this! I have noticed that tests run for new PRs and force-pushed commits, but if a test fails due to a flake I am unable to re-run it (ex: "Run Java PreCommit"). PR that has this issue: https://github.com/apache/beam/pull/10369. Is this intended behaviour? - Kirill On Tue,

Re: [VOTE] Release 2.18.0, release candidate #1

Sorry about the messiness. The links at the bottom should be correct though. I intentionally did not replace MAVEN_VERSION because I didn't know how to get it (I didn't execute mvn for the release). As for JDK_VERSION, do we still need that? (If so, what about Python versions, such as the ones

Re: [BEAM-9015] Adding pyXX-cloud instead of pyXX-gcp and pyXX-aws

now back from the holidays, I intend to do this - one of these days. On Thu, Dec 26, 2019 at 12:51 PM Udi Meiri wrote: > +1 > > On Mon, Dec 23, 2019, 17:28 Robert Bradshaw wrote: > >> Makes sense to me. >> >> On Mon, Dec 23, 2019 at 3:33 PM Pablo Estrada wrote: >> > >> > Hi all, >> > a

Re: Jenkins job execution policy

Does the approval list live beyond the lifetime of the jenkins machine (my initial impression is that the approval list disappears on Jenkins machine restart)? Also, I imagine that ASF wants an explicit way to see who is approved and who is denied which the plugin doesn't seem to allow. On Tue,

Re: Jenkins job execution policy

I've merged https://github.com/apache/beam/pull/10582 to unblock existing contributors that are having trouble getting their PRs tested without committer help. We can discuss Kai's suggestion. Looking at https://github.com/jenkinsci/ghprb-plugin, it seems like the 'add to whitelist' comment adds

Re: [RESULT] [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

I hope for the mascot will be simple enough so everyone can draw it and adapt. The mascot will be license free right... so you don't need to pay the graphic artist for every use of the mascot? _/ _/ Alex Van Boxel On Tue, Jan 14, 2020 at 8:26 PM Aizhamal Nurmamat kyzy wrote: > Thanks Kenn

Re: Jenkins job execution policy

Nice! I took a look at Beam Jenkins job properties ( CommonJobProperties.groovy#L108-L111 ) and it uses jenkinsci/ghprb-plugin

Re: Is Jenkins stuck?

Seems like a global issue, the same was happening on kafka and Hive. ASF Infra restarted the master and the jobs start executing now. https://issues.apache.org/jira/browse/INFRA-19722 https://issues.apache.org/jira/browse/INFRA-19721 On Tue, Jan 14, 2020 at 12:54 PM Luke Cwik wrote: > Stuck

Re: Jenkins job execution policy

I've added all the PR authors for the last 1000 merged PRs. I will merge in a few minutes. I'll have a follow up change to document this on the website. On Tue, Jan 14, 2020 at 11:29 AM Luke Cwik wrote: > Should we scrape all past contributors and add them to the file? > > On Tue, Jan 14, 2020

Re: Is Jenkins stuck?

Stuck for me as well. I tried rerunning them and it did work sometimes. On Tue, Jan 14, 2020 at 12:39 PM Andrew Pilloud wrote: > I've noticed the same thing. The instances I've investigated have actually > ran the tests, but the status never got posted to github. > > On Tue, Jan 14, 2020 at

Re: Is Jenkins stuck?

I've noticed the same thing. The instances I've investigated have actually ran the tests, but the status never got posted to github. On Tue, Jan 14, 2020 at 12:36 PM Boyuan Zhang wrote: > Same here: https://github.com/apache/beam/pull/10375, at least has been > pended for 15 hrs > > On Tue, Jan

Re: Is Jenkins stuck?

Same here: https://github.com/apache/beam/pull/10375, at least has been pended for 15 hrs On Tue, Jan 14, 2020 at 12:31 PM Reuven Lax wrote: > For https://github.com/apache/beam/pull/10534 , all the tests have been > yellow "pending" for the past 15 hours. The same was true before when I >

Is Jenkins stuck?

For https://github.com/apache/beam/pull/10534 , all the tests have been yellow "pending" for the past 15 hours. The same was true before when I tried forcing then all to run again, but the new set of tests is also stuck. Does anyone know what's happening? Reuven

Re: Jenkins job execution policy

Should we scrape all past contributors and add them to the file? On Tue, Jan 14, 2020 at 11:18 AM Kenneth Knowles wrote: > Nice! This will help at least temporarily. We can see if it grows too > unwieldy. It is still unfriendly to newcomers. > > Kenn > > On Tue, Jan 14, 2020 at 11:06 AM Pablo

Re: [RESULT] [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

Thanks Kenn for running the vote! I had reached out to a couple designers that I know personally and few in the community to see whether they were willing to contribute the designs. Julian was one of them, who agreed to work with us for a reasonable fee which can be donated by Google. Julian is

Re: Jenkins job execution policy

Nice! This will help at least temporarily. We can see if it grows too unwieldy. It is still unfriendly to newcomers. Kenn On Tue, Jan 14, 2020 at 11:06 AM Pablo Estrada wrote: > Hi all, > ASF INFRA gave us a middle-ground sort of workaround for this by using > .asf.yaml files. Here's a change

Re: Jenkins job execution policy

Hi all, ASF INFRA gave us a middle-ground sort of workaround for this by using .asf.yaml files. Here's a change to implement it[1], and documentation for the .asf.yaml file[2], as well as the relevant section for our case[3]. I'll check the docs in [2] well before pushing to merge, just to be

Re: GSOC announced!

Great! I will try to propose something for BeamSQL. -Rui On Tue, Jan 14, 2020 at 10:40 AM Pablo Estrada wrote: > Hello everyone, > > As with every year, the Google Summer of Code has been announced[1], so we > can start preparing for it if anyone is interested. It's early in the > process for

GSOC announced!

Hello everyone, As with every year, the Google Summer of Code has been announced[1], so we can start preparing for it if anyone is interested. It's early in the process for now, but it's good to prepare early : ) Here are the ASF mentor guidelines[2]. For now, the thing to do is to file JIRA

Understanding watermark

Hello Beam expert, We have a pipeline (run on Flink) that read both bounded and unbounded sources and our understanding is that when the bounded sources complete they should get a watermark of +inf and then we should be able to take a savepoint and safely restart the pipeline. However, we have

Re: Jenkins jobs not running for my PR 10438

Valentyn, thank you. On Tue, Jan 14, 2020 at 12:05 PM Valentyn Tymofieiev wrote: > > Done. If tests still don't trigger, you could try to make a push to the > branch to reset the test status. > > On Tue, Jan 14, 2020 at 8:38 AM Tomo Suzuki wrote: >> >> Hi Beam developers, >> >> Can somebody

Re: Jenkins jobs not running for my PR 10438

Done. If tests still don't trigger, you could try to make a push to the branch to reset the test status. On Tue, Jan 14, 2020 at 8:38 AM Tomo Suzuki wrote: > Hi Beam developers, > > Can somebody run the following to > https://github.com/apache/beam/pull/10554 ? > Run Dataflow ValidatesRunner >

Re: Jenkins jobs not running for my PR 10438

Hi Beam developers, Can somebody run the following to https://github.com/apache/beam/pull/10554 ? Run Dataflow ValidatesRunner Run Java PreCommit On Mon, Jan 13, 2020 at 2:35 PM Tomo Suzuki wrote: > > Thank you, Mark and Ismaël. > > On Mon, Jan 13, 2020 at 2:34 PM Mark Liu wrote: > > > > done

Re: [DISCUSS] Integrate Google Cloud AI functionalities

Nice idea, IO looks like a good place for them but there is another path that could fit this case: `sdks/java/extensions`, some module like `google-cloud-platform-ai` in that folder or something like that, no? In any case great initiative. +1 On Tue, Jan 14, 2020 at 4:22 PM Kamil Wasilewski <

[DISCUSS] Integrate Google Cloud AI functionalities

Hi all, We’d like to implement a set of PTransforms that would allow users to use some of the Google Cloud AI services in Beam pipelines. Here's the full list of services and functionalities we’d like to integrate Beam with: * Video Intelligence [1] * Cloud Natural Language [2] * Cloud AI

Re: [PROPOSAL] gRPC Vendor Release

Thanks Luke, and good to saw that the vendored Dependencies already Released. Best, Jincheng Luke Cwik 于2020年1月10日周五 上午2:09写道： > Thanks for the work, I'll nominate myself as the release manager for > beam-vendor-grpc-1_26_0. > > On Wed, Jan 8, 2020 at 5:38 PM jincheng sun > wrote: > >> Hi