Re: Streaming pipelines in all SDKs!

it happens! Your friendly neighborhood distributed gopher wrangler, Robert Burke Related: PR 8531 [1] begins adding automates testing of the Go SDK against Flink, which should assist with ensuring this eventual work keeps working. [1]: https://github.com/apache/beam/pull/8531 On Thu, May 9, 2019, 6:32

Re: CVE audit gradle plugin

Fri, Apr 19, 2019 at 10:12 AM Robert Burke wrote: > >> If we move to Go Modules, the go.mod file specifies direct dependencies >> and versions, and the go.sum file includes checksums of the full transitive >> set of dependencies. There's likely going to be a tool for detectin

Re: CVE audit gradle plugin

If we move to Go Modules, the go.mod file specifies direct dependencies and versions, and the go.sum file includes checksums of the full transitive set of dependencies. There's likely going to be a tool for detecting if an update is possible, if one doesn't exist in the go tooling already. On

Re: Go SDK status

> > Cheers, > Nathan > > On Wed, 17 Apr 2019 at 20:27, Lukasz Cwik wrote: > >> Thanks for the indepth summary. >> >> On Mon, Apr 15, 2019 at 4:19 PM Robert Burke wrote: >> >>> Hi Thomas! I'm so glad you asked! >>> >>> The

Re: Python SDK timestamp precision

+1 for plan B. Nano second precision on windowing seems... a little much for a system that's aggregating data over time. Even for processing say particle super collider data, they'd get away with artificially increasing the granularity in batch settings. Now if they were streaming... they'd

Re: New contributor to Beam

Welcome Cyrus! :D Yay better docs! On Wed, 17 Apr 2019 at 10:20, Connell O'Callaghan wrote: > Welcome Cyrus!!! > > On Wed, Apr 17, 2019 at 10:11 AM Mikhail Gryzykhin > wrote: > >> Welcome! >> >> --Mikhail >> >> On Wed, Apr 17, 2019 at 9:58 AM Melissa Pashniak >> wrote: >> >>> >>> Welcome

Re: Go SDK status

tegration tests running against a ULR or Flink (there are Jira's for these). 2. Improving existing IOs, adding tests for existing features over adding new ones. a) Migrate the existing IOs to use the Go CDK where possible (needs to wait for the Versioning/GoModules/Gradle issue to be resolved though).

Re: Go SDK status

Give me another hour. It's not a brief email to write. On Mon, 15 Apr 2019 at 10:43, Pablo Estrada wrote: > +Robert Burke ; ) thoughts? > > - AFAIK, we have wordcount running on Flink > > On Sat, Apr 13, 2019 at 11:31 AM Thomas Weise wrote: > >> How "expe

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

del. > > > This email is already very long, but in summary I think the right > answer is to just get rid of Outer altogether (except possibly for > IOs, which we'd only preserve for legacy reasons until 3.0). The > question remains whether we should allow a Coder C (as part of its >

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

idea why there's a separate copy >>>> for Python ? I didn't see a significant difference in definitions looking >>>> at few random coders there but I might have missed something. If there's no >>>> reason to maintain two, we should probably unify. >>>> Al

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

maintain two, we should probably unify. >> Also, seems like we haven't added the definition for UTF-8 coder yet. >> >> > Not certain as well. I did notice the timer coder definition didn't exist > in the Python copy. > > >> >>> Here is an example PR[3] that adds th

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

String UTF8 was recently added as a "standard coder " URN in the protos, but I don't think that developed beyond Java, so adding it to Python would be reasonable in my opinion. The Go SDK handles Strings as "custom coders" presently which for Go are always length prefixed (and reported to the

Re: [PROPOSAL] Preparing for Beam 2.12.0 release

I'm going to go out on a limb and assume you mean first RC cut on April 3rd, and the Cherry-pick deadline EoD (PST?) April 2nd. On Thu, 28 Mar 2019 at 09:23, Andrew Pilloud wrote: > It seems like there was some confusion around when the branch cut was > going to happen. I cut the branch

Increase Portable SDK Harness share of memory?

A question came over the beam-go slack that I wasn't able to answer, in particular for Dataflow*, is there a way to increase how much of a Portable FnAPI worker is dedicated for the SDK side, vs the Runner side? My assumption is that runners should manage it, and have the Runner Harness side be

Re: Build blocking on

gt;> >>> @Robert >>> As said before, I am a bit scared about the implications. Shelling out >>> is done by python, and from build perspective, this does not work very >>> well, unfortunately. I.e. no caching, up-to-date checks etc... >>> >>> But o

Re: Build blocking on

rspective, this does not work very >>> well, unfortunately. I.e. no caching, up-to-date checks etc... >>> >>> But of course, we need to play with this a bit more. >>> >>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke wrote: >>> >>>> Reading

Re: Build blocking on

Go dep management, over trying to fix GoGradle. On Tue, 26 Mar 2019 at 09:43, Udi Meiri wrote: > Robert, from what I recall it's not flaky for me - it consistently fails. > Let me know if there's a way to get more logging about this error. > > On Mon, Mar 25, 2019, 19:50 Robert

Re: Build blocking on

It's concerning to me that 1) the Go dependency resolution via gogradle is flaky, and 2) that it can block other languages. I suppose 2) makes sense since it's part of the container bootstrapping code, but that makes 1) a serious problem, of which I wasn't aware. I should have time to investigate

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations Mark!  On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev wrote: > Congratulations, Mark! > > Thanks for your contributions, in particular for your efforts to > parallelize test execution for Python SDK and increase the speed of Python > precommit checks. > > On Sun, Mar 24,

Re: [Go SDK] User Defined Coders

s between the proposal and schemas, I'm going to start a PR for this. Cheers, Robert Burke PS. I've added a link to the doc and the other Go specific ones to the Technical/Design Doc page of the wiki <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95653903> to make them eas

Re: [Go SDK] User Defined Coders

> >> >>> >> >> >>> >> >> >>> This is far more readable than what we have today, and what >> unlocks this is that Beam actually knows the structure of the record >> instead of assuming records are uncrackable blobs. >> >> >>> >&g

Re: [Go SDK] User Defined Coders

rst, I think > this will guide us; e.g. we can build up the coder support in a way that > can be extended to full schemas later. > > Writing up an overview design shouldn't take too much time and I think is > definitely worth it. > > Reuven > > On Mon, Jan 7, 2019 at 2:12 PM Robert Burke

Re: Query expressions for schema fields

In the eventual future where the Go SDK supports schemas, it should be possible to use struct Field Tags to specify paths for extraction from schema data, for usage similar to what Java uses parameter Annotations. eg. type MyKey struct { K string `jsonpath:userid` } type MyValue struct{ K

Re: [Go SDK] User Defined Coders

Kenn has pointed out to me that Coders are not likely going to vanish in the next while, in particular over the FnAPI, so having a coder registry does remain useful, as described by an early adopter in another thread. On Fri, Jan 4, 2019, 10:51 AM Robert Burke wrote: > I think you're ri

Re: Beam Contribution

I like that idea. Even something as simple as "I'm working on X,Y,Z and here's my github profile." It would also be handy for allowing users tagging in specific folks if there's a particular aspect they want to discuss. On Mon, Jan 7, 2019, 7:01 AM Maximilian Michels wrote: > Hi David, > >

Re: Schemas in the Go SDK

pproaches to do the same thing. I'm on the side of Scalable Data Processing in Go, which ideally entails writing Go, rather than an abstract DSL. I don't speak for all Go users, and welcome hearing from others. On Thu, 3 Jan 2019 at 17:52 Robert Burke wrote: > At this point I feel like the schema d

Re: [Go SDK] User Defined Coders

quot;, Sum.ofLongs(), >>>>> "total_cost")) >>>>> .aggregateField("purchaseCost", Top.largestLongs(10), >>>>> "top_purchases")) >>>>> >>>>> >>>>> This is far more reada

Re: [Go SDK] User Defined Coders

rovides, but also allows for far more readable code. We can't make such a > change in Beam Java 2.X for compatibility reasons, but maybe in Go we're > better off starting with schemas instead of coders? > > Reuven > > On Thu, Jan 3, 2019 at 8:45 PM Robert Burke wrote: > &g

Schemas in the Go SDK

SideInputs are configured in the Go SDK. I'd be concerned about the efficiency of those operations though, even with Generics or code generation. On Thu, 3 Jan 2019 at 16:33 Reuven Lax wrote: > On Fri, Jan 4, 2019 at 1:19 AM Robert Burke wrote: > >> Very interesting Reuven! >

Re: [Go SDK] User Defined Coders

; that's only the specification half, not the using half. >>> >>> As it stands, the code generator I've been building these last months >>> could (in principle) statically analyze a user's struct, and then generate >>> an efficient dedicated coder for it. It just

[Go SDK] User Defined Coders

atives, or other suggestions and opinions, I'd love to hear them! Otherwise my intent is to get a PR ready by the end of January. Thanks! Robert Burke

Re: Add code quality checks to pre-commits.

I had the same question, and tt supports many more than we do: https://www.sonarqube.org/features/multi-languages/ All the various rules checks have clear explanations and justifications for why they're doing what they do. It would be quite handy as part of the precommits I think, if at least as

Re: ULR Tests on commit?

gt; >> I believe this is in-progress. Dan, do you have a JIRA for tracking? >> >> On Wed, Dec 12, 2018 at 10:08 AM Robert Burke wrote: >> >>> In our auto populated github PR template, we have a variety of SDK >>> languages to runner combos, but the Un

Re: OOO

Have a great bonding time! I'd say "break" but I expect you'll be quite busy. On Wed, Dec 12, 2018, 9:57 AM Etienne Chauchot wrote: > Enjoy your family time and take care of the little one > > Etienne > > Le mardi 11 décembre 2018 à 12:26 +0100, Maximilian Michels a écrit : > > Thank you for

ULR Tests on commit?

. Thanks! Robert Burke @lostluck, distributed gopher wrangler

Re: JB's back

Welcome back! On Wed, Nov 21, 2018, 11:46 AM Rui Wang wrote: > Welcome back! > > -Rui > > On Wed, Nov 21, 2018 at 10:33 AM Pablo Estrada wrote: > >> It's good to have you back JB : ) - hope all is well. >> -P. >> >> On Wed, Nov 21, 2018 at 10:32 AM Ankur Goenka wrote: >> >>> Good to have you

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

+1 to Option 3 I'd rather have each SDK have a single point of well defined complexity to do something general, than have to make tiny but simple changes. Less toil and maintenance in the long run per SDK. Similarly I don't have time to make it happen right now. On Tue, Nov 13, 2018, 9:22 AM

Wiki edit access please

I'd like to add more information about contributing to the Go SDK. Cheers, Robert B

Re: [DISCUSS] Gradle for the build ?

Hi all! Coming from the Go SDK side, I'm in favour of splitting the repo after the portability APIs have stablized. The reason being that it would be all to easy for changes not be propagated to the SDKs, which would cause painful drift. As for the Go Gradle experience, it agree it leaves much to

Resolving Go SDK build/test failures when using gradle

If you haven't run into :beam-sdks-go:buildLinuxAmd64 or similar failing with "undefined: passert.Sum" recently, stop reading now. The root cause is that the gogradle plugin doesn't clean up the vendor directories that it sets up, in combination

Re: Python PreCommit broken

The issue is time from commit to merge, and without manual intervention, commits from other PRs aren't accounted for, if there's a lag between LGTM and merge. On Fri, Sep 21, 2018, 1:52 PM Ahmet Altay wrote: > I will suggest a rollback in this case, and in general as a good practice > to

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

Congrats Kenn! :D On Wed, Sep 19, 2018, 2:21 PM Ismaël Mejía wrote: > Congratulations and welcome Kenn as new chair! > Thanks Davor for your hard work too. > > On Wed, Sep 19, 2018 at 11:14 PM Rui Wang wrote: > >> Congrats! >> >> -Rui >> >> On Wed, Sep 19, 2018 at 2:12 PM Chamikara Jayalath

Re: [portablility] metrics interrogations

The way I entered them into the Go SDK is #2 (SDK sends diffs per bundle) and the Java Runner Harness appears to aggregate them correctly from there. On Mon, Sep 10, 2018, 2:07 AM Etienne Chauchot wrote: > Hi all, > > @Luke, @Alex I have a general question related to metrics in the Fn API: > as

Re: [Proposal] Track non-code contributions in Jira

+1 On Sun, Aug 26, 2018, 7:19 PM Griselda Cuevas wrote: > Thanks all for the votes and comments. > > re:Jira - I hear and understand all questions about Jira not being the > absolute best option, however I think the positive points are strong enough > to still use is as primary option. > > re:

Want to talk Beam at GopherCon?

1.11 means for the SDK + Harness Performance + Usability and documentation (godoc and Beam site) + Coders + Getting some light hacking done maybe? Looking forward to meeting you! Robert Burke

Re: CODEOWNERS for apache/beam repo

+1 If non-committers are welcome in the file, I'm happy to assist Henning with Go SDK reviews.(@lostluck) On Tue, Jul 10, 2018, 8:47 AM Alexey Romanenko wrote: > +1, > Udi, thank you for taking care of this! > I added myself as a reviewer of some IO components. > > > On 10 Jul 2018, at 17:00,

Re: [DISCUSS] Automation for Java code formatting

+1! Given this is the recommended default for Go projects (with it's own gofmt tool) I'm for similar tooling for this in other languages. I suspect we can add Gradle command to run gofmt over the go code too for a consistent hook to run for beam code. It would save folks from needing to set up

Re: [FYI] New Apache Beam Swag Store!

Thank you Matthias! On Wed, Jun 20, 2018, 1:09 AM Matthias Baetens wrote: > Hey Robert, > > I have requested to include this in the webshop or send me the sizes via > e-mail. I'll let you know if I get an update on this! > > Cheers, > Matthias > > On Thu, 14 Jun

Re: [FYI] New Apache Beam Swag Store!

Would it be possible to get the brand of the shirts & sweaters, and the sizing information? The vendor should have that available. The pages have a lot of fabric and cut description, but not the measurements. Sizes fit varies dramatically on the brand, and if I get a hoodie, anything from a Medium

Re: [VOTE] Code Review Process

+1 On Mon, Jun 4, 2018, 9:01 AM Raghu Angadi wrote: > +1 > > On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh wrote: > >> As we seem to largely have consensus in "Reducing Committer Load for Code >> Reviews"[1], this is a vote to change the Beam policy on Code Reviews to >> require that >> >> (1)

Re: Reducing Committer Load for Code Reviews

+1 I also thought this was the norm. My read of the committer/contributor guide was that a committer couldn't unilaterally merge their own code (approval/LGTM needs to come from someone familiar with the component), rather than every review needs two committers. I don't recall a requirement

Re: [VOTE] Go SDK

+1 (non-binding) I'm looking forward to helping gophers solve their big data problems in their language of choice, and runner of choice! Next stop, a non-java portability runner? On Tue, May 22, 2018, 6:08 AM Kenneth Knowles wrote: > +1 (binding) > > This is great. Feels like

Re: I'm back and ready to help grow our community!

Congrats & welcome back! On Thu, May 17, 2018, 5:44 PM Huygaa Batsaikhan wrote: > Welcome back, Gris! Congratulations! > > On Thu, May 17, 2018 at 4:24 PM Robert Bradshaw > wrote: > >> Congratulations, Gris! And welcome back! >> On Thu, May 17, 2018 at

Re: Beam high level directions (was "Graal instead of docker?")

The approach you're looking for sounds like the user's Runner of Choice, would use a user side version of the runner core, without changing the Runner of Choice? So a user would update their version of the SDK, and the runner would have to pull the core component from the user pipeline? That

Re: ValidatesRunner test cleanup

I am curious as to how long the suite takes with the changes you've made. How long does a full Validates Runner suite take with your recategorizing? On Thu, May 3, 2018, 9:24 AM Eugene Kirpichov wrote: > Thanks Scott, this is awesome! > However, we should be careful when

Re: [Go SDK] Proposal: Set up a Vanity Import Path

; > > > On Wed, Apr 18, 2018 at 6:29 PM, Robert Burke <rob...@frantil.com> wrote: > > This seems like enough consensus to file the JIRA, so > > https://issues.apache.org/jira/browse/BEAM-4115 has now been created. > > > > I'll get to work on the PRs shortly.

Re: [Go SDK] Proposal: Set up a Vanity Import Path

This seems like enough consensus to file the JIRA, so https://issues.apache.org/jira/browse/BEAM-4115 has now been created. I'll get to work on the PRs shortly. Cheers, Robert Burke On Wed, 18 Apr 2018 at 03:52 Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > +1 > > Agree >

Re: Merge options in Github UI are confusing

+1 Having made a few web commits and been frustrated by the options, anything to standardize on a single option seems good to me. On Tue, 17 Apr 2018 at 01:49 Etienne Chauchot wrote: > +1 to enforce the behavior recommended in the committer guide. I usually > ask the

[Go SDK] Proposal: Set up a Vanity Import Path

uestions, but please let me know if you have others either here, or in a doc comment. If there's consensus to do so, it would be better it's done sooner rather than after folks begin depending on it. We wouldn't want to have fragmented examples. Robert Burke (One of the Gopher Googlers who have

Re: [ANNOUCEMENT] New Foundation members!

Congratulations! On Sat, 31 Mar 2018 at 11:53 Ekrem Aksoy wrote: > Congrats! > > On Sat, Mar 31, 2018 at 2:08 AM, Davor Bonaci wrote: > >> Now that this is public... please join me in welcoming three newly >> elected members of the Apache Software

<    1   2   3   4   5