Re: [DISCUSS] Autoformat python code with Black

+1 to autoformatters and yapf. It appears to be a well maintained project. I do support making a one time pass to apply formatting the whole code base. On Tue, Jan 21, 2020 at 5:38 PM Chad Dombrova wrote: > >> It'd be good if there was a way to only apply to violating (or at >> least changed)

Re: [DISCUSS] Autoformat python code with Black

> > > It'd be good if there was a way to only apply to violating (or at > least changed) lines. I assumed the first thing we’d do is convert all of the code in one go, since it’s a very safe operation. Did you have something else in mind? -chad > > On Tue, Jan 21, 2020 at 1:56 PM Chad

Re: Updating Metrics Counter in user defined thread

I'm curious in this setup how one attributes the reporting of counters on bundle completion. (Often there is more than one bundle running concurrently in a VM.) Another idea--would it be possible (this would require some change to user code, but not much) to pass a Counter object that's already

Re: [VOTE] Release 2.18.0, release candidate #1

> Also, does anyone know how can I (we) validate the new docker image for Flink's job server included in this release? To start the job server: docker run --net=host apachebeam/flink1.9_job_server:2.18.0_rc1 Then you can run any Beam Java/Python/Go job with job endpoint localhost:8099 to

Re: Updating Metrics Counter in user defined thread

Thanks for the suggestions, Robert and Alex! Although the suggested solutions may work, all of them require users to change their code, which is not ideal. Creating new threads and updating metrics in callback functions are pretty common for samza-runner users. We'd like to come up with a way that

Re: Jenkins jobs not running for my PR 10438

Thank you Rui and Ankur! On Tue, Jan 21, 2020 at 4:14 PM Ankur Goenka wrote: > Done > > On Tue, Jan 21, 2020 at 4:03 PM Kirill Kozlov > wrote: > >> Forgot to paste a link to a PR: https://github.com/apache/beam/pull/10649 >> >> On Tue, Jan 21, 2020 at 3:47 PM Rui Wang wrote: >> >>> Done but I

Re: Jenkins jobs not running for my PR 10438

Forgot to paste a link to a PR: https://github.com/apache/beam/pull/10649 On Tue, Jan 21, 2020 at 3:47 PM Rui Wang wrote: > Done but I am not seeing tests are triggered by those two commands. > > On Tue, Jan 21, 2020 at 3:42 PM Kirill Kozlov > wrote: > >> Hello again! >> >> Could someone

Re: Beam SQL Query Parameters

Nice write up of the options. Noting here that Beam SQL's public API is pretty @Experimental :-). We should probably start thinking about which bits are the most stable and declare them. This new stuff would still be experimental at the method/feature level anyhow. Kenn On Tue, Jan 21, 2020 at

Re: Jenkins jobs not running for my PR 10438

Done but I am not seeing tests are triggered by those two commands. On Tue, Jan 21, 2020 at 3:42 PM Kirill Kozlov wrote: > Hello again! > > Could someone trigger tests on this PR please? > *Run SQL postcommit* > *Run JavaBeamZetaSQL PreCommit* > > On Mon, Jan 20, 2020 at 2:38 PM Kirill Kozlov

Re: Jenkins jobs not running for my PR 10438

Hello again! Could someone trigger tests on this PR please? *Run SQL postcommit* *Run JavaBeamZetaSQL PreCommit* On Mon, Jan 20, 2020 at 2:38 PM Kirill Kozlov wrote: > Thank you, Ismaël! > > On Mon, Jan 20, 2020 at 2:14 PM Ismaël Mejía wrote: > >> done >> >> On Mon, Jan 20, 2020 at 9:27 PM

Beam SQL Query Parameters

Hi all, I want to implement query parameters in Beam SQL, and I hope to get some feedback from others because this involves adding to Beam SQL's public API. https://docs.google.com/document/d/1kPG3S6qAIPlhTYPYK_CYwkzBk-i7U6vON5_95tRGk6c/edit?usp=sharing Thanks, Kyle

Re: [DISCUSS] Autoformat python code with Black

Huge +1 to autoformatters. Yapf solves my biggest complaint about black, that it can't be customized (and I didn't want to take on maintaining our own fork of black). It'd be good if there was a way to only apply to violating (or at least changed) lines. On Tue, Jan 21, 2020 at 1:56 PM Chad

Re: [DISCUSS] Autoformat python code with Black

+1 to autoformatting Let me add some nuance to that. The way I see it there are 2 varieties of formatters: those which take the original formatting into consideration (autopep8) and those which disregard it (yapf, black). I much prefer yapf to black, because you have plenty of options to tweak

Re: [DISCUSS] Integrate Google Cloud AI functionalities

The current state is that it works, and a large amount of testing is being added [1], but the public API is still in flux (especially the java-as-callee side [2], and the specification of dependencies [3,4]). It is being actively worked on though. [1] https://github.com/apache/beam/pull/10051 [2]

Re: [VOTE] Release 2.18.0, release candidate #1

On Tue, Jan 21, 2020 at 12:04 PM Ahmet Altay wrote: > > This change (https://github.com/apache/beam/pull/10625) was merged after the > RC1 email was out. IMO, we do not need to block RC1 vote for this. If there > will be an RC2 the change will be included. Agreed, we do not need to block RC1

Re: PreCommit Java Portability is failing

This is caused by PR 10557 , I will fix it soon. On Tue, Jan 21, 2020 at 1:24 PM Kirill Kozlov wrote: > Hello beam developers! > > Jenkins job page: > https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/ > The following tasks fails: `

Re: [DISCUSS] Autoformat python code with Black

+1 to yapf, I normally run yapf before submitting PR and it seems to be able to solve most of my lint errors (indentations most of the time) with the correct style file. it will be great if yapf has an option to only autoformat the diff change, but seems not officially supported. (I use a vim

PreCommit Java Portability is failing

Hello beam developers! Jenkins job page: https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/ The following tasks fails: ` *:runners:google-cloud-dataflow-java:buildAndPushDockerContainer*` with error: *'command 'docker'' finished with non-zero exit value 1* Is it possible that

Re: Ordering of element timestamp change and window function

On Thu, Jan 16, 2020 at 9:52 PM Kenneth Knowles wrote: > > > On Thu, Jan 16, 2020 at 11:38 AM Robert Bradshaw > wrote: > >> On Thu, Jan 16, 2020 at 11:00 AM Kenneth Knowles wrote: >> > >> > IIRC in Java it is forbidden to output an element with a timestamp >> outside its current window. >> >>

Re: [DISCUSS] Autoformat python code with Black

Last time we discussed this there seems not to be much progress into autoformatting. This tool looks more tweakable, so maybe it could be more appropriate for Beam's use case. https://github.com/google/yapf/ WDYT? On Thu, May 30, 2019 at 10:50 AM Łukasz Gajowy wrote: > +1 for any autoformatter

Re: [VOTE] Release 2.18.0, release candidate #1

If there is a rc2 I would like to get BEAM-9144 since this is a regression reported already by one user. I am not sure however if it counts as a blocker since it is a quite particular use case. Also, does anyone know how can I (we) validate the

Re: [VOTE] Release 2.18.0, release candidate #1

This change (https://github.com/apache/beam/pull/10625) was merged after the RC1 email was out. IMO, we do not need to block RC1 vote for this. If there will be an RC2 the change will be included. I recall we had a similar thread before. Please, include the release managers in the PRs that

Re: [VOTE] Release 2.18.0, release candidate #1

I was not aware of https://issues.apache.org/jira/browse/BEAM-9123 or the PR on the release branch. On Tue, Jan 21, 2020 at 11:16 AM Robert Bradshaw wrote: > The source tarball seems to be missing the commit at > > https://github.com/apache/beam/commit/a61dfbf4570e3adb30e15315c116751faeda897e >

Re: [VOTE] Release 2.18.0, release candidate #1

The source tarball seems to be missing the commit at https://github.com/apache/beam/commit/a61dfbf4570e3adb30e15315c116751faeda897e On Tue, Jan 21, 2020 at 9:49 AM Ahmet Altay wrote: > > All, could you help with validations and voting? > > On Wed, Jan 15, 2020 at 6:14 PM Ahmet Altay wrote: >>

Re: [VOTE] Release 2.18.0, release candidate #1

All, could you help with validations and voting? On Wed, Jan 15, 2020 at 6:14 PM Ahmet Altay wrote: > +1, validated the same things, they still work. Thank you. > > On Wed, Jan 15, 2020 at 5:01 PM Udi Meiri wrote: > >> Dataflow containers have been updated. Test away. >> >> On Tue, Jan 14,

Re: beam_PreCommit_Java_Examples_Dataflow_Phrase is failing

Yes, it was a back port to 2.19. Thank you for helping with that, Boyuan! > On 20 Jan 2020, at 21:00, Boyuan Zhang wrote: > > Are you running the tests against 2.19.0 release branch? If so, it fails for > tests misconfiguration and we have a fix for that: >

Re: [DISCUSS] Integrate Google Cloud AI functionalities

Hello, we are synced I was exactly back to need that same functionality. Last time I checked (end november 2019) there were still many things that were not there. First the External transform is not yet correctly exposed to SDK users (see the previous discussion [1] and Jira ticket BEAM-8546 [2]).

Re: [DISCUSS] Integrate Google Cloud AI functionalities

Is using Python from Java via ExternalTransform working and tested? On Tue, Jan 21, 2020 at 6:50 AM Reza Rokni wrote: > +1 for using cross language transforms. > > On Thu, 16 Jan 2020 at 01:23, Ahmet Altay wrote: > >> >> >> On Wed, Jan 15, 2020 at 8:12 AM Kamil Wasilewski < >>