Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations Mark! 🎉 On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev wrote: > Congratulations, Mark! > > Thanks for your contributions, in particular for your efforts to > parallelize test execution for Python SDK and increase the speed of Python > precommit checks. > > On Sun, Mar 24, 201

Re: Build blocking on

It's concerning to me that 1) the Go dependency resolution via gogradle is flaky, and 2) that it can block other languages. I suppose 2) makes sense since it's part of the container bootstrapping code, but that makes 1) a serious problem, of which I wasn't aware. I should have time to investigate

Re: Build blocking on

ol for handling Go dep management, over trying to fix GoGradle. On Tue, 26 Mar 2019 at 09:43, Udi Meiri wrote: > Robert, from what I recall it's not flaky for me - it consistently fails. > Let me know if there's a way to get more logging about this error. > > On Mon, Ma

Re: Build blocking on

>> is done by python, and from build perspective, this does not work very >>> well, unfortunately. I.e. no caching, up-to-date checks etc... >>> >>> But of course, we need to play with this a bit more. >>> >>> On Tue, Mar 26, 2019 at 6:24 PM Robert Burke

Re: Build blocking on

dm/.gradle/go/repo/cloud.google.com' folder? >>> >>> @Robert >>> As said before, I am a bit scared about the implications. Shelling out >>> is done by python, and from build perspective, this does not work very >>> well, unfortunately. I.e. no caching, u

Increase Portable SDK Harness share of memory?

A question came over the beam-go slack that I wasn't able to answer, in particular for Dataflow*, is there a way to increase how much of a Portable FnAPI worker is dedicated for the SDK side, vs the Runner side? My assumption is that runners should manage it, and have the Runner Harness side be as

Re: [PROPOSAL] Preparing for Beam 2.12.0 release

I'm going to go out on a limb and assume you mean first RC cut on April 3rd, and the Cherry-pick deadline EoD (PST?) April 2nd. On Thu, 28 Mar 2019 at 09:23, Andrew Pilloud wrote: > It seems like there was some confusion around when the branch cut was > going to happen. I cut the branch yesterda

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

String UTF8 was recently added as a "standard coder " URN in the protos, but I don't think that developed beyond Java, so adding it to Python would be reasonable in my opinion. The Go SDK handles Strings as "custom coders" presently which for Go are always length prefixed (and reported to the Runn

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

issed something. If there's no >> reason to maintain two, we should probably unify. >> Also, seems like we haven't added the definition for UTF-8 coder yet. >> >> > Not certain as well. I did notice the timer coder definition didn't exist > in the Python copy.

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

> >>>> Ah great. Thanks for the pointer. Any idea why there's a separate copy >>>> for Python ? I didn't see a significant difference in definitions looking >>>> at few random coders there but I might have missed something. If there's no >>

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

Outer to Inner involves changing the second > component. I don't think we want to bake special rules like this into > the model. > > > This email is already very long, but in summary I think the right > answer is to just get rid of Outer altogether (except possibly for > IOs,

Re: Go SDK status

Give me another hour. It's not a brief email to write. On Mon, 15 Apr 2019 at 10:43, Pablo Estrada wrote: > +Robert Burke ; ) thoughts? > > - AFAIK, we have wordcount running on Flink > > On Sat, Apr 13, 2019 at 11:31 AM Thomas Weise wrote: > >> How "exp

Re: Go SDK status

hem. This should enable others to contribute beam features to the SDK. * The Versioning Issue mentioned above, it's finicky. * Large (State Backed) Iterable Support *What I'd love help with* 1. Getting the existing suite of SDK integration tests running against a ULR or Flink (there are Jir

Re: New contributor to Beam

Welcome Cyrus! :D Yay better docs! On Wed, 17 Apr 2019 at 10:20, Connell O'Callaghan wrote: > Welcome Cyrus!!! > > On Wed, Apr 17, 2019 at 10:11 AM Mikhail Gryzykhin > wrote: > >> Welcome! >> >> --Mikhail >> >> On Wed, Apr 17, 2019 at 9:58 AM Melissa Pashniak >> wrote: >> >>> >>> Welcome Cyrus

Re: Python SDK timestamp precision

+1 for plan B. Nano second precision on windowing seems... a little much for a system that's aggregating data over time. Even for processing say particle super collider data, they'd get away with artificially increasing the granularity in batch settings. Now if they were streaming... they'd probab

Re: Go SDK status

r of last year. > > Cheers, > Nathan > > On Wed, 17 Apr 2019 at 20:27, Lukasz Cwik wrote: > >> Thanks for the indepth summary. >> >> On Mon, Apr 15, 2019 at 4:19 PM Robert Burke wrote: >> >>> Hi Thomas! I'm so glad you asked! >>&g

Re: CVE audit gradle plugin

If we move to Go Modules, the go.mod file specifies direct dependencies and versions, and the go.sum file includes checksums of the full transitive set of dependencies. There's likely going to be a tool for detecting if an update is possible, if one doesn't exist in the go tooling already. On Fri,

Re: CVE audit gradle plugin

ll/8354 > > On Fri, Apr 19, 2019 at 10:12 AM Robert Burke wrote: > >> If we move to Go Modules, the go.mod file specifies direct dependencies >> and versions, and the go.sum file includes checksums of the full transitive >> set of dependencies. There's likely going

Re: Streaming pipelines in all SDKs!

ackling this, whenever it happens! Your friendly neighborhood distributed gopher wrangler, Robert Burke Related: PR 8531 [1] begins adding automates testing of the Go SDK against Flink, which should assist with ensuring this eventual work keeps working. [1]: https://github.com/apache/beam/pull/8531

Re: Intro

Welcome aboard :D On Tue, 14 May 2019 at 13:28, Ahmet Altay wrote: > Welcome! Added you as a contributor to JIRA. > > *From: *Damien Desfontaines > *Date: *Tue, May 14, 2019 at 1:24 PM > *To: * > > Hi folks, >> >> I'm Damien from the Anonymization team at Google. I might contribute a >> couple

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Woohoo! Well deserved. On Tue, May 14, 2019, 8:34 PM Reuven Lax wrote: > Congratulations! > > *From: *Mikhail Gryzykhin > *Date: *Tue, May 14, 2019 at 8:32 PM > *To: * > > Congratulations Pablo! >> >> On Tue, May 14, 2019, 20:25 Kenneth Knowles wrote: >> >>> Hi all, >>> >>> Please join me and

Re: PardoLifeCycle: Teardown after failed call to setup

What is the runner supposed to be doing to trigger the teardown of given bundle descriptors in an SDK harness? Is there a fn API call I'm not interpreting correctly that should reliably trigger DoFn teardown, or generally that bundle processing is done? On Wed, May 15, 2019, 6:51 AM Robert Brad

Re: Writing bytes to BigQuery with beam

For the Go SDK: BigQueryIO exists, but other than maybe one PR that added batching of writes (to avoid the size limit communicating with BigQuery), the reads are probably going to be re-written I don't believe there's any s

Re: [Discuss] Ideas for Apache Beam presence in social media

+1 As a twitter user, I like this idea. On Mon, 20 May 2019 at 15:18, Aizhamal Nurmamat kyzy wrote: > Hello everyone, > > What does the community think of making Apache Beam’s social media > presence more active and more community driven? > > The Slack and StackOverflow for Apache Beam offer pre

Re: Hazelcast Jet Runner

(minor related tangent for additional perspective) +1 from the perspective of SDKs on moving from experimental to production versions being a breaking change. I've long posited that the Go SDK, as it's currently experimental, is v0.X, and some breaking changes have been made accordingly. Once the

Re: Measuring element sizes in benchmarks

The Go SDK doesn't yet have these counters implemented or published (sampling elements &countinf between DoFns, etc). On Tue, May 28, 2019, 9:08 AM Alexey Romanenko wrote: > On 28 May 2019, at 17:31, Łukasz Gajowy wrote: > > > I'm not quite following what these sizes are needed for--aren't the

Re: Question about building Go SDK

Not a bother at all! The Gradle based build for the Go SDK is brittle, and there are a few issues around it. Notably that it's doesn't really line up with how end users will typically acquire dependencies, using the regular go toolchain. As Luke says, if you're working with or *on* the SDK, one ca

Re: I'm thinking about new features, what do you think?

I'm not sure I understand the desired properties of GroupByMultiKey. Offhand, am I right interpreting GroupByMultiKey as essentially forming a graph of the keys based on the MultiKeys nodes, and the number of resulting iterables is based on the components of the graph. If that's the case then, wh

Re: [DISCUSS] Portability representation of schemas

Wouldn't SDK specific types always be under the "coders" component instead of the logical type listing? Offhand, having a separate normalized listing of logical schema types in the pipeline components message of the types seems about right. Then they're unambiguous, but can also either refer to ot

Re: [ANNOUNCE] New committer: Mikhail Gryzykhin

Congrats On Fri, Jun 21, 2019, 12:29 PM Thomas Weise wrote: > Hi, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Mikhail Gryzykhin. > > Mikhail has been contributing to Beam and actively involved in the > community for over a year. He developed the community buil

Re: Golang dependencies in .test-infra/tools

Unfortunately it seems you're already following the right path to get it done. If you haven't cleared the gradled vendor directories and retune install deps, do that. You might simply have Go HEAD deps that we haven't updated the lock file for. The longer term solution is the oft discussed but not

Re: [Discuss] Create stackoverflow tags for python, java and go SDKs?

The new tagging also helps if they identify the runner separately (flink, spark, google-cloud-dataflow, etc.) On Tue, Jul 9, 2019, 10:16 AM Rui Wang wrote: > I also agree `apache-beam` + `python` or `java` is an option. I can try to > tag SO questions by this option when there is a need to see h

Re: [ANNOUNCE] New committer: Robert Burke

2019 at 12:45 PM Łukasz Gajowy >>>>>> wrote: >>>>>> >>>>>>> Congratulations! :) >>>>>>> >>>>>>> śr., 17 lip 2019 o 04:30 Rakesh Kumar >>>>>>> napisał(a): >>>>>

Re: precommits failing on git clean:

First time contributor Zach might have a solution in this PR, but it seems like it would need care since it's pretty broad. https://github.com/apache/beam/pull/9096 On Fri, Jul 19, 2019, 11:53 AM Udi Meiri wrote: > https://issues.apache.org/jira/browse/BEAM-7788 >

Re: How to expose/use the External transform on Java SDK

Ideas inline. On Wed, Jul 24, 2019, 9:56 AM Ismaël Mejía wrote: > After Beam Summit EU I was curious about the External transform. I was > interested on the scenario of using it to call python code in the > middle of a Java pipeline. This is a potentially useful scenario for > example to evaluat

Talk Beam Go at GopherCon?

If anyone wants to talk about the Apache Beam Go SDK, I'm at GopherCon in San Diego this week. Please say hi to the moustached, blue haired gentleman (me). There's no official Beam content on the program, but why should that stop us? Robert Burke (@lostluck on Twitter)

Re: [ANNOUNCE] New committer: Kyle Weaver

Congrats! Also, thanks for getting the Go SDK integration tests running against Flink and Spark as well :D On Wed, Aug 7, 2019, 11:21 PM Rakesh Kumar wrote: > Congrats Kyle!! > > On Wed, Aug 7, 2019 at 11:30 AM Heejong Lee wrote: > >> Congratulations! >> >> On Wed, Aug 7, 2019 at 11:05 AM Tanay

Re: Proposal for SDFs in the Go SDK

Thanks for the spending the time writing this up! I'm looking forward to seeing how the prototype implementation plays out. In particular with the extensive section on how users will actually use the presented API to get their DoFns to scale. (Disclosure: I helped pre-review the document, which i

Re: Allowing firewalled/offline builds of Beam

If the work to switch to using Go Modules under gogradle works, then it should be possible to use a proxy hosted inside the firewall for the go packages, rather than the vendoring directories. On Thu, Aug 8, 2019, 11:17 AM Lukasz Cwik wrote: > Udi beat me by a couple of mins. > > We build a good

Re: [discuss] How we support our users on Slack / Mailing list / StackOverflow

For the Go SDK, emailing the dev list or asking on Slack are probably the best ways to get an answer from me. I'm not in the habit to search for open Go SDK questions on stack overflow right now, but will chip in if they're pointed out to me As Alexey mentions, Slack largely works for quick back a

Re: contributor permission for Beam Jira tickets

Absolutely! What's your Jira username? You can create an account if you don't already have one, following the instructions here: https://beam.apache.org/contribute/#prerequisites I'm here to help with most things Go SDK too, and provide timely reviews and merges of Go PRs. Just be sure to mention

Re: contributor permission for Beam Jira tickets

You should be set to self assign issues. (Thanks Pablo) Welcome! On Fri, Sep 20, 2019, 12:21 PM dev wearebold wrote: > Hey my Jira username is johnpatoch69 > > Thank you > > Regards, > > J > > Le 20 sept. 2019 à 20:57, Robert Burke a écrit : > > Absolutely! W

Re: Why is there no standard boolean coder?

+1 I'm happy to whip together the Go SDK version once the encoding has been concretely decided. On Fri, Sep 27, 2019, 6:07 PM Chad Dombrova wrote: > > It would still be a standard coder - the distinction I'm proposing is that >> there are certain coders that _must_ be implemented by a new runne

Want to talk Beam at GopherCon?

pics: + What Go 1.11 means for the SDK + Harness Performance + Usability and documentation (godoc and Beam site) + Coders + Getting some light hacking done maybe? Looking forward to meeting you! Robert Burke

Re: [Proposal] Track non-code contributions in Jira

+1 On Sun, Aug 26, 2018, 7:19 PM Griselda Cuevas wrote: > Thanks all for the votes and comments. > > re:Jira - I hear and understand all questions about Jira not being the > absolute best option, however I think the positive points are strong enough > to still use is as primary option. > > re: T

Re: [portablility] metrics interrogations

The way I entered them into the Go SDK is #2 (SDK sends diffs per bundle) and the Java Runner Harness appears to aggregate them correctly from there. On Mon, Sep 10, 2018, 2:07 AM Etienne Chauchot wrote: > Hi all, > > @Luke, @Alex I have a general question related to metrics in the Fn API: > as

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

Congrats Kenn! :D On Wed, Sep 19, 2018, 2:21 PM Ismaël Mejía wrote: > Congratulations and welcome Kenn as new chair! > Thanks Davor for your hard work too. > > On Wed, Sep 19, 2018 at 11:14 PM Rui Wang wrote: > >> Congrats! >> >> -Rui >> >> On Wed, Sep 19, 2018 at 2:12 PM Chamikara Jayalath >>

Re: Python PreCommit broken

The issue is time from commit to merge, and without manual intervention, commits from other PRs aren't accounted for, if there's a lag between LGTM and merge. On Fri, Sep 21, 2018, 1:52 PM Ahmet Altay wrote: > I will suggest a rollback in this case, and in general as a good practice > to unblock

Resolving Go SDK build/test failures when using gradle

If you haven't run into :beam-sdks-go:buildLinuxAmd64 or similar failing with "undefined: passert.Sum" recently, stop reading now. The root cause is that the gogradle plugin doesn't clean up the vendor directories that it sets up, in combination

Re: [DISCUSS] Gradle for the build ?

Hi all! Coming from the Go SDK side, I'm in favour of splitting the repo after the portability APIs have stablized. The reason being that it would be all to easy for changes not be propagated to the SDKs, which would cause painful drift. As for the Go Gradle experience, it agree it leaves much to

Wiki edit access please

I'd like to add more information about contributing to the Go SDK. Cheers, Robert B

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

+1 to Option 3 I'd rather have each SDK have a single point of well defined complexity to do something general, than have to make tiny but simple changes. Less toil and maintenance in the long run per SDK. Similarly I don't have time to make it happen right now. On Tue, Nov 13, 2018, 9:22 AM Tho

Re: Contributor status change

Welcome aboard! :D On Fri, Nov 16, 2018, 11:48 AM Kenneth Knowles wrote: > Welcome! I have added you to the "Contributors" role. > > Kenn > > On Fri, Nov 16, 2018 at 11:46 AM Adrian Witas wrote: > >> >> *Hi, my name is Adrian Witas. I am interested in contributing GO SDK to >> the Apache Beam S

Re: JB's back

Welcome back! On Wed, Nov 21, 2018, 11:46 AM Rui Wang wrote: > Welcome back! > > -Rui > > On Wed, Nov 21, 2018 at 10:33 AM Pablo Estrada wrote: > >> It's good to have you back JB : ) - hope all is well. >> -P. >> >> On Wed, Nov 21, 2018 at 10:32 AM Ankur Goenka wrote: >> >>> Good to have you b

ULR Tests on commit?

. Thanks! Robert Burke @lostluck, distributed gopher wrangler

Re: OOO

Have a great bonding time! I'd say "break" but I expect you'll be quite busy. On Wed, Dec 12, 2018, 9:57 AM Etienne Chauchot wrote: > Enjoy your family time and take care of the little one > > Etienne > > Le mardi 11 décembre 2018 à 12:26 +0100, Maximilian Michels a écrit : > > Thank you for you

Re: ULR Tests on commit?

> ULR. >> >> I believe this is in-progress. Dan, do you have a JIRA for tracking? >> >> On Wed, Dec 12, 2018 at 10:08 AM Robert Burke wrote: >> >>> In our auto populated github PR template, we have a variety of SDK >>> languages to runner combos

Re: Add code quality checks to pre-commits.

I had the same question, and tt supports many more than we do: https://www.sonarqube.org/features/multi-languages/ All the various rules checks have clear explanations and justifications for why they're doing what they do. It would be quite handy as part of the precommits I think, if at least as

[Go SDK] User Defined Coders

alternatives, or other suggestions and opinions, I'd love to hear them! Otherwise my intent is to get a PR ready by the end of January. Thanks! Robert Burke

Re: [Go SDK] User Defined Coders

ating a JSON schema from a proto file >>> <https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema>, but >>> that's only the specification half, not the using half. >>> >>> As it stands, the code generator I've been building these last months

Schemas in the Go SDK

e worked out, similarly to how SideInputs are configured in the Go SDK. I'd be concerned about the efficiency of those operations though, even with Generics or code generation. On Thu, 3 Jan 2019 at 16:33 Reuven Lax wrote: > On Fri, Jan 4, 2019 at 1:19 AM Robert Burke wrote: > >>

Re: [Go SDK] User Defined Coders

e schemas > the basic semantics instead of coders. Schemas provide everything a coder > provides, but also allows for far more readable code. We can't make such a > change in Beam Java 2.X for compatibility reasons, but maybe in Go we're > better off starting with schemas ins

Re: [Go SDK] User Defined Coders

t;) >>>>>.aggregateField("purchaseCost", Sum.ofLongs(), >>>>> "total_cost")) >>>>> .aggregateField("purchaseCost", Top.largestLongs(10), >>>>> "top_purchases")) >&

Re: Schemas in the Go SDK

the SDK language, and this moves it farther away from the more general, if verbose approaches to do the same thing. I'm on the side of Scalable Data Processing in Go, which ideally entails writing Go, rather than an abstract DSL. I don't speak for all Go users, and welcome hearing from others.

Re: Beam Contribution

I like that idea. Even something as simple as "I'm working on X,Y,Z and here's my github profile." It would also be handy for allowing users tagging in specific folks if there's a particular aspect they want to discuss. On Mon, Jan 7, 2019, 7:01 AM Maximilian Michels wrote: > Hi David, > > Welco

Re: [Go SDK] User Defined Coders

Kenn has pointed out to me that Coders are not likely going to vanish in the next while, in particular over the FnAPI, so having a coder registry does remain useful, as described by an early adopter in another thread. On Fri, Jan 4, 2019, 10:51 AM Robert Burke wrote: > I think you'

Re: Query expressions for schema fields

In the eventual future where the Go SDK supports schemas, it should be possible to use struct Field Tags to specify paths for extraction from schema data, for usage similar to what Java uses parameter Annotations. eg. type MyKey struct { K string `jsonpath:userid` } type MyValue struct{ K s

Re: [Go SDK] User Defined Coders

pport first, I think > this will guide us; e.g. we can build up the coder support in a way that > can be extended to full schemas later. > > Writing up an overview design shouldn't take too much time and I think is > definitely worth it. > > Reuven > > On Mon, Jan 7, 2019

Re: [Go SDK] User Defined Coders

gt;> "total_cost")) >> >> >>> .aggregateField("purchaseCost", >> Top.largestLongs(10), "top_purchases")) >> >> >>> >> >> >>> >> >> >>> This is far more readable than what we have today,

Re: [Go SDK] User Defined Coders

x27;t currently find any insurmountable issues between the proposal and schemas, I'm going to start a PR for this. Cheers, Robert Burke PS. I've added a link to the doc and the other Go specific ones to the Technical/Design Doc page of the wiki <https://cwiki.apache.org/confluence/pag

Re: Naming! Dataflow Worker/SDK "Harness" image flag

+1 to shorter flags without unnecessary words On Mon, Apr 19, 2021, 3:19 PM Robert Bradshaw wrote: > I commented on the doc, but I'm also in favor of dropping "harness." > > On Mon, Apr 19, 2021 at 3:10 PM Tyson Hamilton wrote: > >> I'm in favor of dropping "harness" and going with "sdk_contain

Re: Error comparing flat schema against schema inferred from protoclass

It looks like it doesn't consider TYPE to match TYPE NOT NULL. I don't know how Beam Java handles that but I'd guess you'd need to annotate somehow the fields to make them match. On Tue, Apr 20, 2021, 12:24 PM Fernando Morales Martinez < fernando.mora...@wizeline.com> wrote: > sure thing! > This

Re: Question about transformOverride

My general answer for this is to avoid bundling the IOs with the rest of the pipeline. Have the Input collection be a parameter to a function that constructs the rest of the pipeline, which returns its intended PCollections as outputs. No need to go as far as wrap the whole construction function a

Re: [ANNOUNCE] New committer: Yichi Zhang

Congratulations Yichi! On Thu, Apr 22, 2021, 7:17 AM Alexey Romanenko wrote: > Congratulations, well deserved! > > On 22 Apr 2021, at 10:03, Jan Lukavský wrote: > > Congrats Yichi! > On 4/22/21 4:58 AM, Ahmet Altay wrote: > > Congratulations Yichi! 📣📣📣 > > On Wed, Apr 21, 2021 at 6:48 PM Chamik

Re: Event time matching of side input

I thought the matching happened with elements in the matching window, in Event time, not in Processing time. Granted, I'm not that familiar with this area myself, but one key part of Beam is nearly everything is Event time by default, not Processing time. On Tue, Apr 27, 2021, 12:43 AM Jan Lukavs

Re: Question about SplittableDoFn

IIRC the Initial Restrictions method gives you an element and you return the restrictions relative to that element. It's entirely appropriate to stat files or query databases in order to determine the initial restrictions and partitions of the data. On Tue, May 18, 2021, 3:21 PM Miguel Anzo Palo

Re: Missing copyright notices due to LICENSE change

ut forth the PR. I'd rather not make the change so close to it, and we can cherry pick it otherwise. Robert Burke Beam Go Busybody On Wed, Mar 24, 2021, 9:25 AM Robert Burke wrote: > I'm less concerned about the Go Doc at this point. > > 0. The Go SDK is still experimental (

Re: Missing copyright notices due to LICENSE change

021 at 9:22 AM Robert Burke wrote: > >> The owners at pkg go.dev say they can't properly recognize the python >> license (see https://github.com/golang/go/issues/45095) due to the >> license being somewhat domineering (a go project could *only* have that >> license if i

[Proposal] Enable Branch Protection for `release-.*` branches

ditional ~3 day period), could a volunteer PMC set up such a protection rule, and prevent my error from recurring? Cheers Robert Burke Beam Go Busybody.

Re: [Proposal] Enable Branch Protection for `release-.*` branches

Ok. I'll fill a ticket with Infra tomorrow once the objection period has passed. Thanks! On Wed, Jun 9, 2021, 12:51 PM Kenneth Knowles wrote: > Great idea. I think only infra can do it. > > On Tue, Jun 8, 2021, 14:27 Robert Burke wrote: > >> During the last branc

[Proposal] Go SDK Exits Experimental

hen it comes to dependency management. It will remove the "+incompatible" tags one sees on the pkg.go.dev list at [4]. I'm very happy to answer any questions you might have about the SDK, and provide additional links as needed. I intentionally avoided a link barrage in this email, as

Re: [Proposal] Enable Branch Protection for `release-.*` branches

Filed with INFRA: https://issues.apache.org/jira/browse/INFRA-21995 On 2021/06/10 15:11:51, Robert Burke wrote: > Ok. I'll fill a ticket with Infra tomorrow once the objection period has > passed. Thanks! > > On Wed, Jun 9, 2021, 12:51 PM Kenneth Knowles wrote: > >

Re: [Proposal] Enable Branch Protection for `release-.*` branches

This is done. We should be unable to delete release branches accidentally. Thanks everyone. On 2021/06/14 19:45:24, Robert Burke wrote: > Filed with INFRA: > https://issues.apache.org/jira/browse/INFRA-21995 > > On 2021/06/10 15:11:51, Robert Burke wrote: > > Ok. I

Re: [Proposal] Go SDK Exits Experimental

p.golang.org/doc/go1.17 does include a Register calling convention, leading to a modest performance improvement across the board. Cheers, Robert Burke On 2021/06/15 18:10:46, Robert Bradshaw wrote: > +1 to declaring Golang support out of experimental once the Go Modules > issues are solved. I don&#

Re: Java precomit failing, (though no test are failing)

Very odd as those paths do resolve now, redirecting to their pkg.go.dev paths. Very odd. This feels transient, but it's not clear why that would return a 404 vs some other error. On Wed, 16 Jun 2021 at 15:39, Kyle Weaver wrote: > For tasks without structured JUnit output, we have to scroll up /

Re: [Proposal] Go SDK Exits Experimental

7:56 PM Kenneth Knowles wrote: > > > > > > +1 awesome > > > > > > On Wed, Jun 16, 2021 at 10:33 AM Robert Burke > wrote: > > >> > > >> Sounds reasonable to me. I agree. We'll aim to get those (Go modules > and LICENSE issue) don

Re: [Proposal] Go SDK Exits Experimental

Regarding documentation update: Initial PR is https://github.com/apache/beam/pull/15057 which goes up to section ~4.3. JIRA link for Programing Guide changes: https://issues.apache.org/jira/browse/BEAM-12513 On 2021/06/17 14:58:54, Robert Burke wrote: > Yup! > > My immediate p

Re: [Proposal] Portable MultimapState

Seems reasonable. Assuming consensus happens, please do add this to the list of design docs on the wiki: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=95653903#content/view/95653903 On Mon, Jun 28, 2021, 4:50 PM Robert Bradshaw wrote: > Thanks. Approach 2 looks reasonab

Re: ProcessContext.timestamp

IIRC it should be the Publish timestamp, not the read timestamp. On Tue, Jul 13, 2021, 2:19 PM Hemali Sutaria wrote: > Beam Developers, > > Is ProcessContext timestamp the same as pubsub's publishTime ? > > https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage > > Or, is it the ti

Re: Missing copyright notices due to LICENSE change

Go SDK docs visible sooner. (This lets the BPG refer to those links reliably as well). Thanks all! On 2021/06/02 01:59:04, Robert Burke wrote: > A link should be fine. The scanner is a similarity detector so small diffs > shouldn't cause issues. > > Though 100% agree

Re: Primitive Read not working with Flink portable runner

On Sun, Jul 25, 2021, 11:09 AM Jan Lukavský wrote: > In general, language-neutral APIs and protocols are a key feature of > portable Beam. > > Yes, sure, that is well understood. But - language neutral APIs requires > language neutral environment. That is why the portable Pipeline > representatio

Re: [ANNOUNCE] New committer: Emily Ye

Congratulations Emily! On Mon, Jul 26, 2021, 4:01 PM Ahmet Altay wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming > a new committer: Emily Ye > > Since joining the Beam community a year ago, Emily has done lots of > contributions across different languages and SDKs,

Re: A simpler way to define and use Java cross-language transforms

Looked at it. LGTM It looks portable enough to be useful for any future Go SDK based Expansion services. I do wonder if there are more general names than "class" but that's a terminology quibble anyway. (Go doesn't use that term, as Go doesn't have inheritance based polymorphism.) Perhaps "type"

Re: Golang: Custom Type Coder

nc jsonDecValue(in []byte) (beam.T, error) { v := Execer{} // ... json unmarshalling code ... return v, nil } ``` Which should get you the desired behavior on distributed runners. Cheers, Robert Burke Beam Go Busybody On 2021/08/01 01:18:40, Tarrence van As wrote: > An addition

Re: Ray-based Apache Beam runner

There's a Runner Authoring Guide on the beam site as well: https://beam.apache.org/contribute/runner-guide/ On 2021/08/11 16:00:32, Robert Bradshaw wrote: > You might also want to look at > https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE/edit#slide=id.p > to

Re: [Proposal] Go SDK Exits Experimental

clear that 2.33.0 is the first non-experimental release (even though that's 4-6 weeks out from actual release.) CHANGES.md will be updated to note the event, but a larger blogpost will happen after the release goes public. Cheers, Robert Burke Defacto Beam Go TL. [1] https://pkg.go.dev/gi

Re: Beam Go SqlTransform API proposal

I've had the opportunity to look at this already, so I've added comments. Thank you for this proposal! On 2021/08/30 18:06:09, Tianyang Hu wrote: > Hi everyone, > > I am a software engineer at Google. I would like to propose a SqlTransform > API in the Beam Go SDK: https://s.apache.org/beam-go

Re: Unexpected in TestStream in Portable Mode

I agree with Robert, having the TestStream be coder Aware seems very strange since it's already followed by a PCollection node that actually knows the type/coder being used. Is there any reason TestStream *needs* to be aware of the type it's bytes segments represent? On Tue, Aug 31, 2021, 10:06 A

Re: Unexpected in TestStream in Portable Mode

Is the claim that the Standard bytes and String_utf8 coders are not "known coders"? What's the point of the standard coders if they are not the canonical "known coders" that can generally be expected to be known by runners/other SDKs? https://github.com/apache/beam/blob/master/model/pipeline/src

Re: Unexpected in TestStream in Portable Mode

;> would require all Runners and SDKs to be updated synchronously for every >>>> new coder they might want to share. >>>> >>>> Instead, what we have are >>>> >>>> * Standard Coders which have well-defined, language-agnostic >>>&g

  1   2   3   4   5   >